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SECTION  I 


INTRODUCTION 


Too  frequently,  textbooks  on  experimental  design  treat 
separately  the  spatial  and  the  temporal  assignments  of  data 
collection  points.  Consequently,  experimenters  more  attentive 
to  the  former  task  often  fail  to  reduce  or  isolate  sources  of 
unwanted  sequence  effects  that  may  severely  bias  experimental 
results  (Poulton,  1975).  In  an  analysis  of  239  experiments 
published  in  Human  Factors  from  1958  to  1972,  Simon  (1976)  found 
that  31%  used  a  form  of  the  "within-subject"  design,  in  which 
each  subject  is  tested  serially  over  many  experimental 
conditions.  Few  of  these,  however,  were  designed  or  analyzed  in 
a  way  that  would  offset  sequence  effects  likely  to  be  engendered 
by  the  design.  > 

Of  the  various  types  of  sequence  effects  that  are  likely  to 
occur,  psychologists  have  shown  the  greatest  concern  for  trends 
associated  with  operator  learning  and  fatigue.  When  such 
effects  are  unwanted,  investigators  have  traditionally  tried  to 
offset  them  by  randomizing  or  systematically  counterbalancing 
the  order  in  which  the  experimental  conditions  are  presented  to 
the  subject.  Unfortunately,  these  methods  have  often  been 
inadequate  and/or  improperly  employed.  Furthermore,  when  large 
scale  multifactor  experiments  are  to  be  designed,  these 
conventional  means  of  handling  sequence  effects  require  more 
replications  of  the  basic  design  than  ordinarily  would  be 
economically  acceptable. 

As  an  alternative,  designs  are  needed  that  are  economical, 
yet  able  to  minimize  the  effects  of  selected  sequence  effects. 
Simon  (1977a)  describes  an  economical  multifactor  2  ^~P  design  of 
Resolution  IV  suitable  for  screening  studies  that  is  robust 
against  linear,  quadratic,  and  cubic  trend  effects  with  no 
replicated  conditions  in  the  design.  In  this  report,  that 
design  is  analyzed  to  determine  to  what  extent  it  is  also  robust 
against  carryover  effects.  A  modified  design  is  proposed  that 
is  robust  against  trend  and  one-trial  carryover  effects. 

WHAT  IS  A  CARRYOVER  EFFECT? 

A  "carryover"  effect  is  said  to  occur  when  experience  on 
one  experimental  condition  affects  the  performance  on  a 
different  condition  that  follows  it  (Simon,  1974).  Carryover 
effects  may  be  simple  or  complex  and  may  last  for  one  or  many 


1 


trials.  In  the  statistical  and  psychological  literature,  such 
terms  as  "change  over",  "residual",  and  "transfer"  have  been 
used  synonymously  with  "carryover",  the  term  that  will  be  used 

in  this  report. 


SECTION  II 


BRIEF  REVIEW  OF  TREND-ROBUST  HOLISTIC  DESIGNS 


In  Table  1,  an  example  of  a  Resolution  IV  trend- 

robust  holistic  design  is  shown.  The  basic  design  is  made  up  of 
32  experimental  conditions.  If  fully  saturated,  it  is 
theoretically  possible  to  examine  the  effects  of  16  different 
factors  (which  may  include  the  "trend  factors") ,  and  to  obtain 
some  indication  of  the  existence  of  critical  two-factor 
interaction  effects.  Main  effects  can  be  estimated 
independently  of  all  other  main  effects  and  of  all  two-factor 
interaction  effects,  but  are  confounded  with  strings  of 
three-factor  interactions  (Appendix  A) .  Two-factor  interactions 
are  aliased  in  sub-sets  of  independent  strings  (Appendix  B) .  As 
with  all  fractional  factorial  designs,  the  indicated  effects  are 
actually  aliased  with  still  higher-order  effects.  Specifically 
in  this  design,  which  is  a  one-two-thousand-and-forty-eighth 
fraction  of  the  full  factorial  design  for  16  factors,  the  main 
effects  are  aliased  with  three-,  five-,  and  all  other 
odd-numbered  interactions  and  the  two-factor  interactions  are 
aliased  with  themselves  and  with  all  even-numbered  interactions. 

This  aliasing  of  sources  of  variance  can  be  tolerated 
in  most  experiments  because  of  certain  characteristics  of  human 
performance  data  and  of  the  paradigm  associated  with  the 
holistic  approach  that  differentiates  it  from  the  traditional 
fractional  factorial  design.  Critical  intrinsic  three-factor 
interactions  are  found  infrequently  in  human  performance  data, 
at  least  when  the  variables  are  quantitative,  and  finding  a 
critical  intrinsic  four-factor  interaction  effect  would  be  an 
extremely  rare  event  (Simon,  1971;  1973;  1976).  Therefore, 

these  designs  can  be  used  with  impunity  as  long  as  the 
assumption  that  higher-order  interactions  are  not  likely  to 
exist  is  employed  only  as  a  tentative  "working  hypothesis"  that 
eventually  will  be  tested  as  prescribed  by  the  holistic 
approach . 

Economy  is  an  important  feature  of  the  holistic  approach. 
Data  for  the  full  factorial  will  probably  never  be  collected 
because  to  do  so  adds  no  new  information  beyond  a  certain  level 

*A11  future  footnotes  are  presented  collectively  beginning  on 
page  42.  They  are  noted  by  the  numbers  in  brackets,  e.g.,  [Ij. 
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of  complexity.  Instead,  data  is  systematically  collected  a 
little  at  a  time  according  to  a  special  paradigm,  each  new  step 
being  taken  only  when  tests  indicate  that  additional  data  is 
likely  to  produce  additional  information.  Some  features 
employed  to  support  the  successful  application  of  this 
heirarchial  approach  are; 

(a)  Data  are  collected  only  to  satisfy  the  requirements  of 

each  step  of  an  informational  heirarchy.  Major  plateaus  in  the 
data  collection  process  include:  (1)  screening  many  potentially 
critical  factors,  (2)  developing  a  response  surface  based  on  the 
critical  ones,  and  (3)  obtaining  more  precise  performance  data 
for  selected  configuration  (see  Simon,  1977b;  1979) . 

(b)  An  elaborate  preexperimental  analysis  is  used  to  try  to 
rationally  spotlight  potentially  critical  interactions  prior  to 
data  collection  and  to  place  them  within  the  fractional 
factorial  experimental  design  where  they  may  best  be  detected  if 
the  assumptions  are  valid. 

(c)  Special  techniques  are  employed  to  detect  the  lack  of 
fit  of  linear  and/or  quadratic  models  (when  continuous  data  are 
involved).  Data  is  not  collected  if  these  tests  fail. 

(d)  Transformation  techniques  are  employed  to  try  to 
eliminate  higher-order  effects.  If  the  model  can  be  linearized 
through  proper  scaling,  the  requirement  for  additional  data  may 
be  eliminated. 

(e)  During  screening,  critical  interactions  are  important 
only  to  the  extent  they  may  involve  factors  that  individually 
did  not  prove  to  be  critical.  When  critical  interaction  strings 
are  detected  it  may  be  necessary  to  collect  additional  data  in  a 
systematic  modular  fashion  to  identify  which  interactions  in  the 
string  are  the  critical  ones  (see  Daniel,  1976). 

In  practice,  one  seldom  uses  a  saturated  experimental 
design.  There  are  practical  reasons  why  the  design  in  Table  1 
—  with  its  potential  for  handling  16  factors  —  would  only  be 
used  to  investigate  no  more  than  approximately  12  factors.  One 
needs  the  extra  degrees  of  freedom  to  represent  sequence  effects 
and  possibly  some  three-factor  interactions. 

Note  that  in  Table  1,  two  sets  of  labels  are  provided:  (1) 
new  screening  design  labels,  and  (2)  original  factor  labels. 

The  first  set  of  labels  is  used  in  an  economical,  multifactor 
experiment  that  takes  into  consideration  the  fractional  nature 
of  the  design,  while  the  second  set  —  the  traditional  one  —  is 
more  useful  when  thinking  about  and  manipulating  the  design 
itself.  Thus,  the  sign  matrix  for  a  2^~P  fractional  factorial 
is  the  same  as  a  2n  factorial  when  (k-p)  =  n.  However,  certain 
advantages  can  be  realized  if  the  columns  are  shifted  about  as 
shown  in  Table  1  for  reasons  to  be  explained  below. 
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The  columns  and  row  of  the  traditional  factorial  design  are 
ordinarily  written  in  Yates'  standard  order.  When  labelled, 
this  order  for  the  columns  would  be  an  expansion  of  the  form 
(original  labels);  A,  B,  AB,  C,  AC,  BC,  ABC,  D,  AD...  and  so 
forth,  depending  on  how  many  factors  are  involved  and  a  similar 
pattern  for  rows;  (1),  a,  b,  ab,  c,  ac  ...  and  so  forth.  In 
this  standard  order,  the  pattern  of  minus  and  plus  signs  in  the 
factor  A  column,  for  example,  would  be  ordered;  -+-+-+-+ 
and  so  forth,  where  the  plus  and  minus  signs  are  associated  with 
experimental  conditions  containing  the  low  and  high  levels  of 
Factor  A,  respectively  (or  any  two  categorical  levels) .  Except 
for  the  need  to  maintain  the  validity  of  the  generalized 
interactions,  which  columns  were  selected  for  main  effects  had 
never  been  an  issue  until  Daniel  and  Wilcoxin  (1966)  showed  how 
it  mattered  and  how  it  could  be  used  to  reduce  the  bias  from 
trend  effects.  Simon  (1977a)  developed  this  procedure  still 
further  and  arrived  at  the  holistic  trend-robust  design  shown  in 
Table  1. 

To  optimize  the  fractional  factorial  so  that  it  will  be 
effectively  robust  against  trend  effects,  the  design  in  Table  1 
is  arranged  as  follows; 

(1)  All  columns  with  letter  A  in  the  original  factor  labels 

are  used  for  the  main  effects.  When  there  are  not  enough  main 

effects  to  fill  the  A-columns,  those  columns  left  over  represent 
independent  strings  of  three-factor  interactions  (Appendix  A) . 

(2)  For  both  main  and  interaction  effects,  the  columns  are 
ordered  (using  original  factor  labels)  so  that  the  highest 
interactions  come  first,  followed  by  smaller  and  smaller 
interactions  until  the  main  effects  are  reached. 

(3)  Within  a  series  of  interactions  of  the  same  degree,  the 
interactions  are  listed  alphabetically  using  the  original 
factorial  labels. 

The  degree  to  which  each  column  of  the  design  is  affected 

by  trend  effects  is  shown  at  the  bottom  of  Table  1  in  the  rows 

labelled;  "Percent  Trend/Effect  Overlap."  This  percent  overlap 
is  the  proportion  of  variance  (times  100)  of  the  factor  effect 
that  could  be  attributed  to  the  trend  effect.  The  column 
rearrangement  per  se  does  not  affect  the  degree  of 
trend-robustness  of  each  column,  but  does  facilitate  the  use  of 
the  overall  design.  For  all  practical  purposes,  no  main  effect 
(new  factor  labels)  will  be  seriously  affected  by  a  linear, 
quadratic,  and/or  cubic  trend  should  any  exist.  Only  five  of 
the  15  two-factor  interaction  strings  will  be  confounded  with 
trends  to  any  notable  effect  and  of  these,  only  three  appear 
serious  enough  to  avoid.  Of  course,  the  importance  of  this 
confounding  also  depends  on  the  relative  strength  of  the  trend 
effects  themselves.  For  example,  a  71%  trend-factor  overlap  may 
be  irrelevant  if  the  actual  trend  effect  is  trivial,  while  a 


modest  10%  overlap  may  be  more  serious  i£  the  trend  itself  is 
very  large.  Rather  than  speculate  too  much  on  the  relative 
magnitudes  of  these  effects,  use  of  the  holistic  design  makes  it 
possible  to  minimize  any  serious  confounding  trends  by: 

(1)  Using  the  results  of  a  preexperiment  analysis  to  assign 
factors  to  the  columns  of  the  design  in  such  a  way  —  to  the 
extent  possible  —  that  potentially  large  main  and  generalized 
interaction  effects  will  be  least  confounded  with  trends. 

(2)  Employing  statistical  techniques  to  remove  the  trend 
components  when  they  cannot  otherwise  be  avoided  (Simon,  1977a}. 

The  importance  of  this  trend^robust  design  is  in  its  power 
and  economy  when  used  for  screening  purposes.  The  most  likely 
sources  of  critical  effects  are  kept  relatively  free  from  trend 
bias.  It  is  not  necessary  to  replicate  and  counterbalance  the 
complete  design  to  achieve  this  as  with  the  conventional 
approach,  which  handles  only  linear  trends  well.  Thus,  a  single 
design  provides  trend-robust  data  for  a  large  number  of  factors 
and  with  a  reliability  that  compares  favorably  with  the  more 
conventional  few-factor'-at-^a'^time  experiments.  These  benefits 
increase  as  the  size  of  the  design  increases  (Simon,  1979). 

The  biggest  drawback  of  the  design  is  that  all  two-^factor 
interactions  are  aliased,  a  maximum  of  eight  in  a  single, 
interaction  string  in  our  design  (Appendix  B) ,  Other 

two-level  Resolution  IV  fractional  factorial  designs  can  be 
found  that  may  have  less  aliasing  among  the  two-factor 
interactions  or  "aberrations  patterns"  (Fries  and  Hunter,  1978), 
but  often  show  some  serious  trend  confounding.  As  in  the  design 
of  any  experiment,  the  investigator  must  compare  the  advantages 
and  disadvantages  of  alternative  data  collection  plans  for  his 
particular  problem  at  the  particular  stage  of  his  investigation. 
The  trend-robust  design  is  intended  first  as  an  economical  means 
of  screening  a  very  large  number  of  factors  to  find  the  truly 
important  ones  and,  as  such,  compares  most  favorably  with  other 
two-level  fractional  factorials  (3)  . 

If  and  when  the  empirical  evidence  suggests  that  critical 
interaction  effects  exist,  the  holistic  approach  provides  an 
economical  data  collection  strategy  for  isolating  them.  It  also 
provides  a.means  of  determining  whether  or  not  the  linear  model 
of  these  designs  adequately  fits  the  data. 

Simon  (1977a,  1977b,  1981)  has  discussed  the  design, 
analysis,  and  interpretation  of  these  trend-robust  designs. 
Westra,  Simon,  Collyer,  and  Chambers  (1981)  successfully 
employed  the  holistic  design  to  evaluate  the  relative  importance 


of  11  features  of  a  pilot  training  simulator  at  the  Naval  Training 
Systems  Center,  Orlando,  Florida.  The  basic  design  was  used  sub¬ 
sequently  in  other  experiments  at  that  facility. 


SECTION  III 


CONCERN  WITH  CARRYOVER  EFFECTS 


E.C.Poulton  at  the  Applied  Psychology  Unit,  Medical 
Research  Council,  Cambridge,  England,  has  identified  a  number  of 
sources  of  bias  in  experimental  data  when  a  subject  is  tested 
across  experimental  conditions.  Among  these  —  in  addition  to 
trend  effects  —  are  range  effects  (1973)  ,  asymmetric  transfer 
(1969),  and  others  (1982)  found  in  cognitive,  perceptual-motor, 
vigilance,  and  psychophysical  judgment  tasks.  Of  particular 
interest  here  is  his  discussion  of  effects  that  transfer,  or 
carryover,  from  one  condition  to  another. 

Unlike  trend  effects  that  tend  to  affect  performance  across 
conditions  over  the  term  of  the  experiment,  carryover  effects 
occur  between  adjacent  conditions  being  studied  sequentially. 
Their  existence  depends  more  on  the  nature  of  the  task  and  the 
characteristics  of  the  conditions  being  tested  than  on  the 
overall  amount  of  practice.  When  a  subject  is  being  tested 
serially  across  conditions,  performance  on  one  condition  may 
affect  performance  on  the  conditions  that  follow. 

Characteristics  of  these  specific  conditions  determine  the 
nature  of  the  carryover  effect,  which  may  be  positive  or 
negative,  additive,  proportional,  or  interactive,  and  may 
persist  beyond  a  single  trial  (Finney,  1956) . 

Carryover  effects  are  task  specific  and  preexperiment 
analysis  is  needed  to  determine  whether  a  particular  design  can 
adequately  handle  them.  There  are  several  types  [4] ,  the 
characteristics  of  which  affect  their  degree  of  persistence. 

For  example,  the  carryover  effect  may  persist  for  only  a  trial 
or  two  when  conditions  differ  in  the  required  magnitude  of  a 
response,  as  would  be  the  case  when  there  are  variations  in  the 
gear  ratio  of  a  steering  wheel.  Carryover  may  extend  over  a 
longer  period  when  one  condition  involves  a  "natural  response" 
(e.g.,  right  movement  of  stick  results  in  a  right  movement  of 
the  display  indicator,  which  a  lifetime  of  experience  has 
reinforced)  that  is  qualitatively  different  from  the  other 
condition,  an  "unnatural  response"  (e.g.,  a  right  movement  of 
the  stick  results  in  a  left  movement  of  the  display  indicator). 
Principles  of  transfer  of  training  frequently  apply  in  these  two 
types  of  carryover.  On  the  other  hand,  there  are  situations  in 
which  the  carryover  may  never  dissipate  once  a  new  strategy  or 
cognitive  principle  has  been  learned  while  performing  on  one 


condition.  Instead,  it  will  persist  and  affect  performance  on 
all  conditions  for  the  remainder  of  the  experiment. 

Counterbalanced  wi thin-subject  Latin  square  designs  have 
been  used  by  psychologists  who  wish  to  reduce  the  presence  of 
individual  differences  in  the  error  variance  and  to  compensate 
for  learning  and  other  sequence  effects.  As  the  most  primitive 
case,  the  2x2  balanced  Latin  square  has  been  employed, 
involving  two  groups  of  subjects,  two  conditions,  and  two  trials 
(with  a  different  condition  on  each  trial  and  each  group 
beginning  with  a  different  condition).  Thus: 

Trial  1  Trial  2 

Group  I  A  B  +  £ 

Group  II  B  A  +  b 

The  capital  letters,  A  and  B,  represent  direct  effects 
while  the  small  letters,  a  and  b,  represent  the  amount  carried 
over  from  the  previous  trial  and  condition. 

As  originally  conceived,  a  2  x  2  Latin  square  is  actually  a 
two-dimensional  representation  of  a  three-dimensional  space. 

The  three  independent  factors  are  Identified  with  the  columns, 
rows,  and  diagonals.  The  2x2  Latin  square  of  our  example  is 
actually  a  fractional-factorial  design,  a  2^’*  plan  with  the 
defining  contrast:  [I]  =  ABC  (see  Simon,  1973  for  further 
explanation  of  this  symbology) .  The  sources  associated  with  the 
three  degrees  of  freedom  show  the  following  aliasing  structure; 
(A  +  BC) ,  (B  +  AC) ,  and  (C  +  AB) .  Interaction  ABC  is  lost  due 
to  the  fractionation.  To  obtain  unbiased  estimates  of  A,  B,  and 
C,  all  interaction  effects  must  be  —  not  merely  assumed  to  be 
—  negligible. 

In  deciding  whether  it  is  appropriate  or  not  to  use  the  2  x 
2  Latin  square  as  a  within-group  design  as  psychologists  do,  it 
helps  to  identify  what  has  been  confounded  and  to  decide  whether 
these  sources  are  likely  to  exist  in  our  experiment.  If  the 
rows  (A),  columns  (B) ,  and  diagonals  (C)  are  labelled  Groups, 
Trials,  and  Conditions,  respectively,  then: 

Designated  Source  Actual  Source  of  Variance 

Conditions  [Conditions  +  (Groups  x  Trials)] 

Groups  [Groups  +  (Conditions  x  Trials)] 

Trials  [Trials  +  (Groups  x  Conditions)] 

The  sources  within  the  brackets  are  totally  aliased  and 
cannot  be  isolated  within  the  context  of  this  design.  In  many 


human  performance  experiments,  these  interaction  terms  represent 
the  following  sources  of  variance: 

Groups  X  Trials;  Groups  with  different  learning  rates 

Conditions  x  Trials;  Conditions  which  are  learned  at 

different  rates 

Groups  X  Conditions:  Groups  with  different  amounts  of 

prior  experience  on  a  condition 

While  experiments  do  differ,  still  it  is  believed  that 
interactions  of  these  types  are  found  too  frequently  in  human 
performance  experiments  to  just  assume  them  out  of  existence. 
Without  the  support  of  directly  relevant  empirical  evidence,  it 
would  be  risky  to  assume  the  additive  model  is  the  correct  one. 
If  it  is  not,  this  2x2  Latin  square  design  should  not  be  used. 

But  problems  with  this  design  do  not  stop  with  these 
potential  interactions.  There  are  also  carryover  effects  that 
can  appear  when  a  wi thin-subj ect  design  is  used.  In  the  2x2 
Latin  square  design  shown  above,  it  can  be  seen  that  the  two 
carryover  effects,  a  and  b,  are  not  balanced  in  the  design. 

Each  does  not  occur  with  all  conditions  of  the  other  dimensions; 
ergo,  carryover  effects  are  not  orthogonal  to  the  other  sources 
of  variance.  Should  there  be  critical  carryover  effects,  this 
situation  will  bias  the  results.  For  screening  purposes,  -it 
would  be  unacceptable  if  direct  condition  effects  were  biased. 

Poulton  and  Freeman  (1966;  Poulton,  1969)  noted  that  if 
amounts  carried  over  from  conditions  A  and  ^  were  equal  —  they 
called  this  "symmetrical  conditions  carryover"  —  then  the 
Condition  effect  would  not  be  biased.  While  symmetrical 
carryover  would  change  the  means  of  the  two  conditions,  the 
difference  between  their  means  (i.e.,  the  Condition  effect) 
would  be  unaffected  since  the  same  amount  is  being  added  or 
subtracted.  However,  Poulton  collected  evidence  to  show  that  in 
many  experiments,  the  size  of  the  carryover  effect  differs 
depending  on  whether  B  follows  A,  or  A  follows  B.  This 
"asymmetrical  transfer,"  they  postulate,  biases  our  estimate  of 
the  condition  effect  and  seriously  jeopardizes  the  effectiveness 
of  within-subject  designs.  This,  they  contend,  argues  for  the 
use  of  between-subj ect  designs  as  the  primary  data  collection 
method  for  most  human  performance  experiments. 

It  helps  our  understanding  of  the  problem  if  we  identify 
Poulton's  symmetrical  and  asymmetrical  carryover  in  terms  of  the 
sources  of  variance  in  the  Latin  square  design.  Symmetrical 
carryover  is  equivalent  to  trial  effects  and  asymmetrical 
carryover  is  an  interactive  carryover  effect.  In  other  words, 
when  conditions  change  the  same  amount  on  the  second  trial, 
regardless  of  the  conditions  involved,  that  is  a  trial  effect 
(if  positive,  possibly  a  learning  effect).  When,  however,  the 


conditions  show  different  amounts  of  carryovetr  we  may  have 
either  an  "additive"  or  an  "interactive"  carryover  effect.  In 
the  former  case,  condition  A  —  while  having  a  different  amount 
of  carryover  than  condition  B  —  always  shows  the  same  amount 
regardless  of  what  condition  follows  it,  while  in  the  latter 
case,  how  much  carryover  A  shows  depends  on  which  condition 
follows  it.  These  distinctions  become  somewhat  blurred  in  the 
oversimplified  2x2  Latin  square,  but  are  important  to  make 
when  larger  Latin  squares  or  other  counterbalanced  designs  are 
involved. 

While  the  evidence  that  Poulton  has  accumulated  shows  that 
the  2x2  Latin  square  wi thin-subj ect  design  as  used  by 
psychologists  definitely  should  not  be  used,  it  does  not  follow 
that  all  within-subj ect  designs  should  be  avoided  in  the 
presence  of  learning  and  carryover. 

ALTERNATIVE  WITHIN-SUBJECT  DESIGNS 

There  are  different  Latin  squares  and  other  balanced 
arrangements  that  can  be  used  to  offset  certain  carryover 
effects,  although  some  are  not  necessarily  economical  nor 
satisfactory  for  our  purposes  (Simon,  1974).  Of  particular 
interest  in  this  discussion,  however,  is  the  "extra-period  Latin 
square"  design  (Simon,  1974,  p.  43-50?  1979,  p.  93-98).  If 

an  investigator  wished  to  correct  the  lack  of  orthogonality  in 
the  2x2  Latin  square  relevant  to  carryover  effects,  an  . 
extra-period  Latin  square  design  might  be  employed.  This  is 
done  by  giving  each  group  a  third  trial,  and  repeating  the 
condition  in  trial  two,  thus; 

Trials 

12  2 

1)  A  B  +  a  B  +  b 

Groups  " 

2)  B  A+b  A  +  2 

In  the  basic  Latin  square  design,  only  one  condition,  B, 
follows  condition  A,  and  vice  versa.  In  this  design,  however, 
conditions  A  and  B  both  follow  conditions  A  and  B. 

Consequently,  each  carryover  condition  is  paired  with  each 
direct  condition  an  equal  number  of  times,  making  these  two 
sources  of  variance  orthogonal.  The  extra-degrees  of  freedom 
obtained  from  the  extra  trials  enables  the  additive  carryover 
effect  to  be  isolated.  The  design  assumes  only  one-trial 
carryover  and  is  not  suitable  if  direct  and  carryover  effects 
interact  or,  for  that  matter,  if  any  of  the  main  effects 
interact . 

A  more  powerful  and  cleaner  design  when  using  a  single 
subject  to  study  the  difference  in  performance  between  two 
conditions  without  concern  for  trend  and  carryover  effects  is 


shown  here.  A  subject  would  be  tested  on  the  two  conditions  in 
the  sequence  shown  below,  with  the  first  A-condition  in 
parentheses  serving  as  a  preconditioner  to  create  a  carryover 
effect  for  the  second  condition.  Performance  on  the 
preconditioner  would  not  be  used  in  the  analysis. 

[5] 

(A)ABBAABBAABBAABBAABBAABBAABBAABBA 


With  this  plan,  the  estimated  Condition  effect,  i.e.,  the 
difference  between  the  means  of  conditions  A  and  B  in  this 
sequence,  will  be  independent  of  additive  and  interactive 
one-trial  carryover  effects  as  well  as  linear,  quadratic,  and 
cubic  trend  effects  of  any  magnitude  running  through  the  data. 
All  six  sources  of  variance  —  direct  and  additive  and 
interactive  carryover  effects  along  with  linear,  quadratic,  and 
cubic  trends  —  each  with  one  degree  of  freedom,  can  be  isolated 
for  all  practical  purposes  leaving  a  healthy  error  term  with  25 
degrees  of  freedom  if  no  blocking  is  used.  Longer  and  shorter 
designs  are  also  available  and  they  can  be  blocked  if  time 
limitations  or  other  restrictions  require  it.  [6] 

While  the  above  examples  are  interesting,  they  are  of 
limited  use  since  few  real-world  problems  can  be  investigated 
adequately  by  comparing  only  two  conditions  (although  it  is 
commonly  done).  Even  the  last  phase  of  an  experimental  effort, 
frequently  the  act  of  comparing  the  effectiveness  of  a  newly 
chosen  configuration  against  that  of  an  older  version,  could 
more  effectively  employ  a  multifactor  approach.  There  are 
almost  always  variations  in  the  context  (including  the 
environment  and  operator  ability)  in  which  two  real-world 
conditions  will  be  operating,  making  the  experimental  question 
of  interest:  "Under  what  conditions  is  A  better  or  worse  than 
B?"  rather  than:  "Is  A  or  B  better  (under  one  experimental 
condition)?"  The  former  question  seeks  a  more  generalizable 
answer  which  can  be  achieved  only  with  a  multifactor  experiment. 
Later,  the  principles  used  to  develop  the  single-factor, 
two-level,  trend-carryover  robust  design  shown  above  will  be 
applied  to  the  development  of  large  multifactor  holistic 
designs. 

BETWEEN-SUBJECT  DESIGN  AS  AN  ALTERNATIVE 


Concern  for  sequence  effects  in  within-subject  designs  has 
led  some  investigators  to  propose  that  between-subj ect  designs 
be  used  exclusively,  or  at  least  in  conjunction  with 
within-subject  designs  to  verify  their  results  (Poulton,  1975; 
1982)  . 

But  between-subj ect  designs  have  their  own  special 
problems,  some  of  which  have  been  noted  by  Poulton  (1975).  He 
writes  (p.  25):  "No  one  likes  separate  groups,  because 

comparisons  between  conditions  are  confounded  by  differences 
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between  the  groups 
for  the  individual 
people.  You  are  1 
you  would  need  for 
you  have  accepted 
tests  take  care  of 
of  the  null  hypoth 
are  not  likely  due 
groups , " 


.  To  ensure  a  reasonable  small  standard  error 
differences,  you  have  to  use  large  numbers  of 
ikely  to  need  more  people  in  each  group  than 
a  complete  within-subject  design.  But  once 
the  need  for  more  people,  your  statistical 
the  differences  between  groups.  A  rejection 
esis  indicates  that  the  obtained  differences 
to  the  chance  allocation  of  people  to 


Using  many  subjects,  however,  increases  the  cost  of  the 
experiment  and  when  large,  multifactor  experiments  are  to  be 
performed,  one  may  run  out  of  time,  money,  and  available 
subjects.  It  may  not  be  logistically  possible  to  use  a 
between-subj ect  design  when  a  great  many  factors  are  to  be 
screened . 

Furthermore,  when  a  large  number  of  replications  (i.e., 
number  of  subjects  within  cells  in  the  between-subj ect  designs) 
are  required  to  obtain  a  small  enough  standard  error  to  overcome 
the  effects  of  individual  differences,  reducing  that  problem 
creates  another.  If  an  investigator  can  afford  the  luxury  of  a 
large  number  of  subjects,  although  in  practice  he  may  not,  he 
will  find  himself  in  the  paradoxical  situation  noted  by  Meehl 
(1967),  that  the  process  of  replicating  to  increase  the 
precision  of  the  experimental  data  decreases  the  confidence  with 
which  one  can  generalize  the  results  to  the  operational 
situation.  While  using  many  subjects  per  group  does  decrease 
the  error  variance  by  a  factor  of  the  square  root  of  the 
quantity  four  over  the  number  of  replications  (Simon  and  Westra, 
1984) ,  it  also  increases  the  chance  of  finding  the  experimental 
data  "statistically  significant"  when  in  fact  the  results  in 
practice  are  negligible  (Bakan,  1966). 

Nor  can  we  assume  that  a  between-subject  design  is  devoid 
of  sequence  effects.  Those  created  by  the  equipment  and  the 
environment  can  still  be  present  and  they  can  distort  operator 
performance.  If,  for  example,  temperature  shifts  affect  the 
sensitivity  of  some  electronic  sensing  device  that  feeds 
information  to  the  operator,  the  visual  signals  may  vary  in  a 
systematic  fashion  throughout  the  day  if  proper  controls  are  not 
taken,  causing  a  comparable  shift  in  performance  among 
operators.  Similarly,  failure  to  allow  for  enough  recycling 
time  for  certain  devices  may  result  in  an  equipment  carryover 
effect  that  confounds  the  performance  of  a  subject  tested  too 
soon  after  the  previous  subject  has  finished.  Sequence  effects 
may  even  be  introduced  by  the  experimenter  who  tests  all  of  the 
subjects,  e.g.,  as  he  becomes  fatigued  or  as  he  gradually  alters 
the  instructions  as  the  experiments  drag  on. 

A  more  subtle  weakness  of  the  between-subject  design  stems 
from  the  "situation  specific"  nature  of  all  human  performance 
experiments.  Results  from  these  experiments  are  a  function  of 


the  characteristics  present  at  the  time  data  is  being  collected, 
whether  studied  or  not,  whether  known  or  not.  If  the 
experimental  subjects  are  not  representative  of  the  population 
to  which  the  results  are  to  be  generalized,  the  information, 
e.g.,  which  factors  critically  influence  the  performance  under 
investigation,  can  be  incorrect.  Subject  skill  level  is  one  of 
the  more  difficult  experimental  factors  to  measure  accurately 
(Simon  and  Westra,  1984)  and  frequently  little  effort  is  made  to 
measure  it  at  all.  The  random  assignment  of  individuals  to  the 
cells  of  a  between-subject  design  does  not  resolve  this  problem 
and  assigning  all  subjects  at  a  specific  skill  level  is 
difficult  to  do  or  for  that  matter  even  to  ascertain  that  it  has 
been  done  correctly.  Introducing  skill  level  as  a  dimension  can 
reduce  this  problem  but  increases  experimental  costs 
considerably  in  large  multifactor  investigations.  The  problem 
is  handled  systematically  and  economically  when  within-subject 
designs  are  used,  since  a  single  naive  subject  can  be  tested 
repeatedly  across  all  the  experimental  conditions,  enabling  the 
investigator  to  see  the  shift  in  results  that  takes  place  as 
ability  improves  with  the  practice. 

In  practice,  the  choice  of  using  a  within-subject  or 
between-subjects  design  will  probably  seldom  rest  solely  on  the 
existence  of  sequence  effects.  Logistical  considerations,  such 
as  subject  availability  and  time  and  money,  are  also  common 
limiting  factors.  Still,  as  Poulton  has  implied,  the  overriding 
principle  in  any  experiment  should  be  to  maintain  the  integrity 
of  the  data.  This  should  mean  minimizing  sequence  effects  and 
all  other  sources  of  irrelevant  variance.  It  should  mean 
including  enough  subjects  to  overcome  otherwise  insurmountable 
sampling  problems.  At  times,  however,  this  may  best  be  achieved 
by  developing  new  procedures  that  will  satisfy  these  multiple 
criteria,  i.e.,  to  obtain  clean,  accurate,  and  general! zable 
information  at  minimal  costs.  When  a  large  multifactor 
experiment  is  being  conducted,  the  opportunities  for  meeting 
these  criteria  increase  over  what  is  possible  with  a  few-factor 
experiment.  A  data  collection  plan  for  that  purpose  will  be 
described  in  the  next  section. 
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SECTION  IV 


CARRYOVER  CHARACTERISTICS  OF  HOLISTIC  DESIGNS 


To  determine  how  resistant  elements  of  the  trend-robust 
holistic  design  are  to  one-trial  carryover  effects,  correlations 
must  be  made  among  the  main  and  interaction  columns  for  direct 
and  for  additive  and  interactive  carryover  effects  as  well  as 
for  linear,  quadratic,  and  cubic  trends.  For  this  analysis  a 
2®  factorial  design  was  used  as  the  test  vehicle.  [7] 

DIRECT  AND  CARRYOVER  SIGN  PATTERNS  OF  AN  EXTRA  PERIOD  2^  DESIGN 

The  96  columns  (with  32  signs  to  a  column)  required  to 
analyze  the  2®  factorial  for  its  trend-carryover  robust 
characteristics  are  created  in  four  sections  as  follows; 

(1)  Direct  Effect  Columns .  In  the  conventional  way,  create 
the  32-row  by  31-column  sign  matrix  of  estimable  main  and 
interaction  direct  effects  for  the  2®  factorial.  Columns 
(effects)  and  rows  (conditions)  will  remain  in  Yates'  standard 
order  for  this  analysis.  No  Identity  column  is  included.  Plus 
or  minus  signs  are  used  to  indicate  which  of  the  two  levels  of 
each  factor  are  found  in  each  experimental  condition  (row). 

These  arithmetic  symbols,  +  and  -,  are  the  conventional 
abbreviated  forms  of  the  labels  +1  and  -1,  respectively. 

(2)  Additive  Carryover  Columns .  Create  a  second  32-row  by 
31-coiumn  sign  matrix  representing  the  additive  carryover 
columns,  one  for  each  of  the  direct  effects  created  previously. 
With  the  experimental  conditions  (rows)  of  the  basic  design 
arranged  in  Yates'  standard  order,  additive  carryover  columns 
are  formed  by  associating  with  each  sign  in  the  direct  effects 
column  the  sign  that  immediately  precedes  it  in  the  columnar 
sequence . 

Since  a  preceding  condition  is  needed  to  create  a  carryover 
effect,  and  since  the  first  row  of  a  2^  matrix  has  no  preceding 
condition,  it  is  necessary  to  add  an  extra  condition  before  the 
first  row  of  the  design,  a  "preconditioner."  This  extra 
condition  should  be  the  same  as  the  one  in  the  last  row  of  the 
basic  2®  design. 

(3)  Interactive  Carryover  Columns.  Create  a  32-row  by 
31-column  sign  matrix  representing  the  interactive  carryover 
columns.  In  each  row,  multiply  the  signs  —  using  the  Rule  of 
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Signs  —  in  corresponding  direct  and  carryover  columns  obtained 
in  the  two  previous  steps. 

(4)  Trend  Columns .  Create  still  another  32-row  by  3-column 
sign  matrix  made  up  of  the  integer  Tchebycheff  orthogonal 
polynomials  (found  in  many  statistical  tables,  e.g.,  in  Beyer, 
1966)  representing  the  linear,  quadratic,  and  cubic  trends 
effects  for  N  =  2^  conditions  of  the  basic  design. 

To  illustrate  the  above  procedures,  the  table  below  was 
created  for  just  two  factors,  A  and  B,  each  at  two  levels.  With 
two,  two-level  factors,  a  4-row  by  12-column  matrix  would  be 
required,  plus  the  "preconditioner,"  P,  thus: 


Additive  Interactive 

Direct  Carryover  Carryover  Trends 


A 

•  B 

AB 

A' 

B' 

AB' 

A" 

B" 

AB" 

L 

Q 

P)[  + 
a)  - 

+ 

+  J 
+ 

+ 

+ 

+ 

+ 

-3 

+1 

b)  + 

- 

- 

- 

- 

+ 

- 

+ 

- 

-1 

-1 

c)  - 

+ 

- 

+ 

- 

- 

- 

- 

+ 

+1 

-1 

d)  + 

+ 

- 

+ 

- 

- 

+ 

- 

+3 

+1 

The  extra-condition,  i.e.,  the  preconditioner,  can  be 
thought  of  as  a  warm-up  trial  and  will  not  be  included  in  the 
analysis.  The  fact  that  the  number  of  conditions  is  not  enough 
to  provide  the  degrees  of  freedom  needed  to  estimate  all  twelve 
effects  is  not  relevant  at  this  time.  For  the  moment,  we  wish 
only  to  illustrate  how  such  a  table  is  constructed. 

CORRELATIONS  AMONG  THE  DIRECT  AND  CARRYOVER  COLUMNS 

Correlations  were  calculated  among  all  96  main  and 
interaction  columns  for  the  31  direct,  31  additive  carryover,  31 
interactive  carryover,  and  3  trend  effects.  From  these 
correlations,  the  main  effect  columns  can  be  selected  that  will 
create  the  best  trend/carryover-robust  design  available  without 
collecting  additional  data. 

The  actual  intercolumn  correlations  are  given  in  Appendix 
C.  However,  to  facilitate  both  the  discussion  and 
interpretation  of  this  rather  large  matrix,  the  full  93-row  by 
96-coiumn  intercorrelation  matrix  was  divided  into  31  x  31 


submatrices  and  three  31  x  3  submatrices,  these  sections  being 
identified  by  letters  of  the  alphabet  in  Table  2. 


TABLE 

2.  SECTIONING  OF 

THE  COMPLETE 

INTERCORRELATION 

MATRIX 

31 

Direct 

31  Additive 
Carryover 

31  Interactive 
Carryover 

3  Trend 

31  Direct 

A 

B 

C 

J 

31  A. 

Carryover 

(D) 

E 

F 

K 

31  I. 

Carryover 

(G) 

(H) 

I 

L 

The  correlations  in  all  sections  of  Table  2  need  not  be 
shown  since  certain  relationships  are  known  on  logical  or  other 
grounds.  For  example,  correlations  in  Sections  D,  G,  and  H  are 
the  same  as  those  in  their  counterparts  B,  C,  and  F, 
respectively.  Only  the  latter  will  be  discussed. 

In  Section  A,  i.e.,  the  intercorrelation  matrix  among 
direct  main  and  interaction  effects  of  a  basic  2®  design,  all 
correlations  on  the  matrix  diagonal  are  1.00,  since  each  direct 
effect  is  correlated  with  itself.  All  off-diagonal  correlations 
are  zero,  meaning  there  is  no  confounding  among  the  direct 
effects,  main  or  interaction,  a  result  to  be  expected  in  an 
orthogonal  factorial  design.  This  section  will  not  be  shown. 

In  Section  E,  i.e.,  the  intercorrelation  matrix  among 
additive  carryover  main  and  interaction  effect  columns,  the 
correlations  again  form  a  unit  matrix,  with  a  diagonal  of  ones 
and  all  off-diagonals  zero.  Since  the  sign  patterns  of  the 
additive  carryover  columns  are  the  same  as  those  of  the  direct 
columns,  except  that  they  are  shifted  down  one  row  and  the  last 
row  cycled  to  the  top,  this  shift  will  not  change  the 
correlations  from  those  found  in  Section  A.  Thus,  the  additive 
carryover  effects  in  Section  E  are  orthogonal  to  one  another. 
This  section  will  not  be  shown  nor  discussed. 

The  correlations  among  linear,  quadratic,  and  cubic  trend 
effects  are  not  included  in  Table  2  since  the  orthogonal 
polynomials  used  to  represent  trend  are,  by  definition, 
uncorrelated.  However,  how  trends  correlate  with  the  direct  and 
the  additive  and  interactive  carryover  columns  are  shown  in 
Table  2  and  will  be  discussed. 

CODING  THE  SYMBOLIC  MATRICES.  Omitting  Sections  A,  D,  E,  G,  and 
H  for  the  reasons  cited  above  leaves  only  Sections  B,  C,  F,  I, 
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J,  K,  and  L  to  be  considered.  To  facilitate  the  study  of  this 
rather  large  matrix  of  numbers,  the  correlations  in  the  matrices 
were  assigned  to  one  of  five  categories  according  to  size  and 
estimated  impact  on  the  interpretation  of  experimental  results 
and  shown  symbolically.  These  categories  r-*  qualitative 
judgments  on  the  part  of  the  investigator  —  were  selected  to 
represent  how  much  a  correlation  within  each  size  range  is 
likely  to  distort  the  interpretation  of  experimental  results 
prior  to  any  effort  to  partial  out  unwanted  effects.  The 
correctness  of  these  judgments  depend,  of  course,  not  only  on 
the  size  of  the  correlation  but  also  on  the  actual  and  relative 
sizes  of  the  effects  being  correlated  and  the  practical 
consequences  of  erroneous  conclusions.  The  distortions 
expected,  therefore,  tend  to  represent  the  worst  case.  It  is 
the  responsibility  of  the  investigator  to  determine  during  the 
preexperiment  analysis  phase  what  conditions  are  likely  to  be 
expected  and  if  they  can  be  tolerated. 

The  five  categories  as  defined  below  are  represented  by  the 
indicated  geometric  symbol; 


Correlation*  Proportion  Overlap  Symbol  Distortion 


0+ 

0 

to  .31 

.0000 

.00  to  .09 

□ 

□ 

B 

None  (N) 

Trivial  (T) 

32 

to  .  54 

.10  to  .29 

Marginal  (M) 

55 

to  .70 

.30  to  .49 

n 

Dangerous  (D) 

>  . 

71 

>  .50 

u 

■ 

Unacceptable  (U) 

*  Correlations  with  30  degrees  of  freedom  (N  =  32)  must 
exceed  .349  to  be  statistically  significant  at  the  p  =  .05 
level.  In  this  report,  correlation  signs  are  ignored. 


Symbolic  representation  of  the  intercorrelation  matrices 
for  Sections  B  and  K,  Sections  C  and  J,  and  Sections  I  and  L, 
are  given  in  Tables  ^’-A,  3'rB,  and  3«'C  respectively.  Because  the 
intercorrelation  matrix  for  Section  F  proved  to  be  identical  ^to 
that  for  Section  C  in  size,  if  not  in  sign  •-  it  is  not 
printed  separately.  It  is  obtained  from  Table  3~B  when  primes 
are  added  to  the  direct  column  labels,  identifying  them  also  as 
additive  carryover  labels. 

The  columns  in  Tables  3-A,  3-s'B,  and  3-C  are  identified  both 
by  their  original  factor  labels  and  the  new  labels  of  a 
trend-rrobust  design.  The  former  represent  the  labels  for  all 
main  and  interaction  colums  of  a  2^  factorial  design.  The 
latter  are  the  labels  for  the  main  and  two-factor  interaction 
strings  of  a  2-26--2J  fractional  factorial  design  used  to  create 
the  trenJ-robust  holistic  design  in  Table  1.  Direct,  additive 
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TABLE  3.  SYMBOLIC  MATRICES  OF  INTERCORRELATIONS  AMONG  DIRECT,  CARRYOVER  AND 
TREND  EFFECT  COLUMNS.  (2*  factorial  or  equivalent  fractional  factorial  design) 


A.  DIRECT  vs  ADDITIVE  CARRYOVER  (Section  B  »  D.  with  K) 


0  r..00»,o.3l|T)  r  •  .55  10 .70  (0) 

H  r  •  .31  to  .54  (M)  H  r  •  >.70  lU) 


ABCOE' 


TABLE  3.  SYMBOLIC  MATRICES  OF  INTERCORRELATIONS  AMONG  DIRECT.  CARRYOVER  AND 
TREND  EFFECT  COLUMNS.  (2’  factorial  or  equivalent  fractional  factorial  design) 

(CONTINUED) 

B.  INTERACTIVE  CARRYOVER  vs  DIRECT*  (Section  C  =  G,  w/ith  J)  _  _ _ _ 

•The  correlations  between  interactive  carryover  and  additive  carryover  effect  columns  (Section  F“  HI  correspond  to  those  shown  in  this 
table  of  correlations  between  interactive  carryover  and  direct  effect  columns,  respectively, except  for  occesional  differences  In  signs. 


Original 

Factor 

Labtlt  New 
Factor 
Labels 
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TABLE  3.  SYMBOLIC  MATRICES  OF  INTERCORRELATIONS  AMONG  Dl  RECT,  CARRYOVER  AND 
TREND  EFFECT  COLUMNS.  (2*  factorial  or  equivalent  fractional  factorial  design) 

(CONTINUED) 


C.  INTERACTIVE  CARRYOVER  vs  INTERACTIVE  CARRYOVER  (Section  I,  with  L) 
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carryover,  and  interactive  carryover  columns  are  distinguished 
by  the  use  of  primes,  none  for  direct,  one  for  additive 
carryover,  and  two  for  interactive  carryover.  For  example,  with 
either  original  or  new  factor  labels,  the  letter  A  would 
represent  the  direct  column  of  factor  A,  A'  the  additive 
carryover  effect  column,  and  A"  the  interactive  carryover 
column.  The  same  use  of  primes  applies  to  interaction  effect 
columns.  When  the  nominal  labels  of  an  interaction  column 
actually  represents  a  string  of  interactions,  the  label  is 
followed  by  a  plus  sign,  e.g.,  AB+.  The  2n  and  S-^factor 
interaction  aliases  for  each  interaction  column  of  Table  1  are 
given  in  Appendices  A  and  B. 

Summary  discussions  of  these  tables  will  be  given  in  the 
next  section  when  the  best  columns  are  selected  in  order  to 
create  a  viable  trendi-carryover  robust  (TCR)  design. 
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SECTION  V 


DEVELOPING  THE  TREND-CARRYOVER  ROBUST  DESIGN 


The  correlation  data  presented  in  the  previous  section  were 
used  to  develop  a  viable  TCR  design.  The  rational  employed  and 
the  results  obtained  are  described  below.  Columns  were  sought 
which  would  result  in  a  reasonably  orthogonal  relationships 
among  direct,  additive  carryover,  interactive  carryover,  and 
trend  effect  columns.  In  cases  where  columns  remained 
correlated  to  a  degree,  it  was  necessary  to  select  the  ones  that 
would  maximize  the  number  of  trivial  relationships,  accept  as 
few  as  possible  marginal  relationships,  and  avoid  dangerous  and 
unacceptable  ones.  In  making  compromises  regarding  these 
selections,  priorities  were  given  to  those  favoring  the 
orthogonality  of  main  effects  over  interaction  effects,  direct 
effects  over  additive  and  interactive  carryover  effects,  and 
direct  with  their  own  additive  and  interactive  carryover  effects 
over  direct  with  other  additive  and  interactive  carryover 
effects.  Tables  3-A,  3-B,  and  3-C  were  Inspected  with  these 
criteria  in  mind. 

CORRELATIONS  AMONG  DIRECT  COLUMNS  (SECTION  A)  AND  AMONG 
ADDITIVE  CARRYOVER  COLUMNS  (SECTION  E) 

As  indicated  earlier,  the  intercorrelation  matrix  among 
direct  columns  as  well  as  the  Intercorrelation  matrix  among 
additive  carryover  columns  are  both  unit  diagonal  matrices. 

Each  column  is  orthogonal  to  all  other  columns  within  the  same 
matrix . 

CORRELATIONS  BETWEEN  DIRECT  AND  ADDITIVE  CARRYOVER 
COLUMNS  (SECTION  B) 

Inspection  of  Table  3-A  (Section  B)  shows  that  the  greatest 
amount  of  nonorthogonality  lies  along  the  diagonal  of  the 
matrix.  This  diagonal  contains  the  correlations  between  each 
direct  column,  whether  main  or  interaction  effect,  and  its  own 
additive  carryover  counterpart.  The  search  was  made  for  columns 
that  were  most  trend  robust  and  had  the  smallest  correlations 
between  themselves  and  their  own  carryover  columns. 

It  has  been  established  that  direct  columns  with  an  A-term 
in  their  original  factor  label  are  the  most  trend  resistant. 
These  columns  have  new  factor  labels  A  through  P  (see  Table  1) . 
Inspecting  the  diagonal  values  for  these  columns  in  Table  3-A, 
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we  find  that  13  out  of  the  31  correlations  between  direct  and 
its  own  additive  carryover  column  are  dangerously  or 
unacceptably  high.  Nine  others  of  these  diagonal  pairs  are 
correlated  marginally  and  nine  trivially,  or  not  at  all. 

If  we  remove  from  consideration  all  direct  main  effect 
columns  that  show  dangerously  or  unacceptably  high  correlations 
with  their  own  additive  carryover  columns  (i.e.,  E,  I,  J,  K,  N, 
0,  and  p) ,  nine  columns  remain  that  tend  to  be  relatively  robust 
against  additive  carryover  effects.  These  are  direct,  main 
effect  columns  with  new  factor  labels:  A,  B,  C,  D,  F,  G,  H,  L, 
and  M.  To  include  their  generalized  interactions  in  the  design, 
we  must  also  select  Interaction-^str ing  columns  labeled:  AB-t, 
AC+,  AD+,  AF+,  AG+,  AH+ ,  AL+,  and  AM+  (see  Table  1). 

Further  inspection  of  Table  3^A  shows  that  four  of  these 
selected  direct  main  effect  columns  (i.e..  A,  C,  F,  and  M)  are 
marginally  correlated  with  their  own  main  and  certain 
interaction»string  additive  carryover  columns.  Direct  effects  A 
and  C  overlap  their  additive  carryover  counterparts  only  14%;  F 
and  M  overlap  theirs  25%.  Of ftrdiagonal  correlations  are  all 
trivial  or  zero  except  for  those  between  M  and  F' ,  M  and  AD+',  F 
and  AD+,  and  F  and  M' ,  all  of  which  are  marginally  correlated. 
Also,  pairs  D  and  AM-t*  and  AM4-  and  D'  are  correlated  at 
unacceptably  high  levels. 

A  more  serious  problem  exists,  however,  with  the  direct 
interaction-»str ing  columns.  Six  out  of  eight  of  these  correlate 
dangerously  or  unacceptably  high  with  their  own  additive 
carryover  interaction-string  columns.  What  appears  to  be  a  good 
selection  for  main  effects  is  poor  insofar  as  their  generalized 
interaction-^strings  are  concerned. 

Under  these  circumstances,  if  a  useful  TCR  design  is  to  be 
developed,  we  must  be  satisfied  that  the  following  statement  is 
viable : 


Pi rect  interaction-strings  have  no  critical 
additive  carryover  effects. 

If  this  hypothesis  is  in  fact  invalid,  then  there  would  be 
no  way  to  determine  whether  a  critical  interaction-string  effect 
is  a  consequence  of  some  interaction  within  the  string  or  the 
string's  additive  carryover  effect.  In  practice,  however,  this 
dilemma  is  probably  a  moot  one,  particularly  if  the  design  is 
used  in  a  screening  experiment.  For  one  thing,  critical 
interaction-strings  in  a  large  multifactor  experiment  are 
infrequent  (though  not  impossible).  That  their  carryover 
effects  might  be  critical  seems  to  be  even  less  likely  in  view 
of  their  composite  nature.  More  pertinent  is  the  fact  that  in 
screening  experiments,  precise  estimations  of  interaction 
effects  are  of  less  concern  than  the  detection  that  they  are 
likely  to  have  critical  effects  on  performance.  For  screening 
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purposes,  we  are  interested  in  critical  interactions  only  to  the 
extent  that  they  contain  a  factor  that  was  not  found  to  be  an 
important  main  effect.  This  condition  occurs  with  intrinsic 
interactions. 

A  casual  check  on  the  validity  of  the  hypothesis  may  be 
made.  One  can  see  how  the  estimated  effects  change  before  and 
after  the  additive  carryover  effects  are  partialled  out  of  the 
direct  main  and  interaction  effects.  If  the  issue  is  still  not 
resolved,  the  proper  procedure  would  be  to  collect  more  data,  a 
necessary  step  anyway  to  determine  which  interactions  within  a 
string  are  actually  contributing  to  any  critical  string  effects. 

It  is  interesting  to  note  that  with  the  exception  of  column 
M,  the  other  remaining  eight  columns  all  have  an  AB~term  in 
their  original  factor  label.  Column  M,  new  label,  is  column  AC, 
original  label.  Furthermore,  these  columns  (i.e..  A,  B,  C,  D, 

F,  G,  H,  and  L,  but  not  M)  form  a  complete  2  factorial. 

CORRELATIONS  BETWEEN  DIRECT  AND  INTERACTIVE  CARRYOVER  COLUMNS 
(SECTION  C)  AND  BETWEEN  ADDITIVE  CARRYOVER  AND  INTERACTIVE 
CARRYOVER  COLUMNS  (SECTION  F) 

The  correlation  matrices  for  Sections  C  and  F  are  identical 
except  for  certain  sign  reversals  so  only  one  matrix  is  given. 
When  developing  the  TCR  design,  magnitude  rather  than  sign  is 
the  critical  concern  regarding  correlations;  therefore, • signs 
are  only  shown  in  the  complete  correlation  table  in  Appendix  C. 

Inspection  of  Table  3^B  (representing  both  Sections  C  and 
F)  along  the  diagonal  reveals  that  between  any  direct  column  and 
its  own  interactive  carryover  column,  the  correlation  is  zero. 
This  is  understandable,  since  the  cross-»product  between  a  direct 
column  and-  its  own  interactive  carryover  column  produces  the 
sign  pattern  of  their  corresponding  additive  carryover  column 
which  has  a  mean  of  zero.  Of f-*diagonal  sign  patterns  reveal  an 
interesting  nonsymmetrical  distribution  of  correlation  values. 
Ninety-two  percent  of  the  possible  900  of f*rdiagonal  correlations 
are  trivially  related  or  not  at  all.  Less  than  3%  are  at  a 
dangerous  or  unacceptable  level.  The  remaining  5%  are  at 
marginal  levels. 

Off-diagonal  correlations  between  the  interactive  carryover 
columns  and  the  nine  main  effect  columns  under  consideration, 
along  with  their  eight  generalized  interaction^-string  columns, 
are  for  the  most  part  trivial  or  zero.  Column  pairs  F  and  F' ,  M 
and  M' ,  and  AD+  and  AD+'  are  each  marginally  correlated  (14% 
overlap)  with  two  interactive  carryover  interaction  columns, 

AC+"  and  AH+";  so  are  columns  F' ,  M' ,  and  AD+'  of  the 
correspondinng  matrix  (not  shown)  of  Section  F.  Only  column  L 
of  Table  3-B  and  column  L*  of  the  corresponding  matrix  of 
Section  F  show  an  uncomfortably  large  number  of  marginal  or 
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dangerous  correlations  with  nine  interactive  carryover  main  or 
interaction  effect  columns. 

CORRELATIONS  AMONG  INTERACTIVE  CARRYOVER  COLUMNS  (SECTION  I) 

A  special  situation  exists  in  Table  3-C  due  to  the  fact 
that  interactive  carryover  column  P"  (new  label  —  the  product 
of  columns  P  and  P'  -’t  contains  only  the  minus  level  condition, 
i.e.,  P"  =  [-1],  the  identity  column.  Consequently,  the  "P" 
effect  cannot  be  estimated.  Furthermore,  when  an  interactive 
carryover  column  containing  an  A-term  in  its  original  factor 
label  is  correlated  with  some  other  interactive  carryover 
column,  the  correlation  obtained  will  be  the  same  as  the 
corresponding  column  without  an  Airterm  in  the  label.  For 
example,  the  size  of  the  correlation  between  any  column  and  ABC” 
will  be  the  same  as  with  that  same  column  and  BC”. 

Inspection  of  the  selected  columns  A, B ,C ,  D,F  ,G  ,H , L ,  and 

M,  and  the  columns  of  their  generalized  interactions  in  Table 
3-C  (Section  I)  shows  that  overall,  intercorrelations  among  the 
interactive  carryover  main  effect  columns  are  generally 
dangerously  or  unacceptably  high.  This  is  not  too  surprising 
since  the  two  levels  (plus  and  minus)  of  a  great  many  of  these 
interactive  carryover  columns  are  not  equally  represented.  This 
less  than  optimum  feature  exists  because,  we  must  remember,  we 
are  examining  the  characteristics  of  an  existing  design,  not  one 
that  has  been  created  specifically  to  study  carryover  effects. 

For  the  select  group,  most  interactive  carryover  main 
effect  columns  are  trivially  or  marginally  correlated  with 
interactive  carryover  interaction  columns.  There  are 
exceptions.  Column  AM+"  is  correlated  dangerously  or 
unacceptably  high  with  seven  out  of  the  nine  interactive 
carryover  main  effect  columns  and  D"  is  dangerously  correlated 
with  column  AL+", 

Too  many  correlations  among  the  interactive  carryover 
interaction  columns,  as  a  group,  are  dangerously  or  unacceptably 
high. 


Because  of  the  serious  confounding  among  interactive 
carryover  main  effect  columns,  another  limitation  must  be  placed 
on  the  use  of  this  TCR  design,  i.e.: 

Interactive  carryover  effects  will  not  be  estimated  with 
this  TCR  design . 

This  does  not  seriously  affect  the  use  of  this  design  as  a 
screening  tool  where  the  primary  purpose  is  to  determine  which 
factors  are  having  critical  effects  on  performance.  The 
original  reason  for  developing  a  TCR  design  was  not  to  estimate 
carryover  effects  but  to  isolate  them  from  direct  effects  as 
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much  as  possible.  It  does  tend  to  accomplish  this  to  a  greater 
degree  than  more  conventional  designs  tend  to  do. 

CORRELATIONS  OF  DIRECT  AND  CARRYOVER  COLUMNS  WITH  TRENDS 
(SECTIONS  J,  K,  AND  L) . 

The  degree  o£  correlation  between  trends  and  direct, 
additive  carryover,  and  interactive  carryover  effect  columns  are 
found  at  the  bottom  of  Tables  3-tB,  3-A,  and  3-C,  respectively. 
Inspection  reveals  that  while  the  direct  main  effects  are  at 
most  only  , trivially  correlated  with  linear,  quadratic,  and  cubic 
trends,  the  direct  interaction-strings  have  some  very  high  and 
also  marginal  relationships  distributed  among  the  trends.  This 
had  already  been  shown  in  Table  1.  This  means  that  in  practice 
one  must  judiciously  assign  the  factors  to  the  appropriate 
columns  in  a  way  that  is  likely  to  avoid  serious  confounding 
with  trends  when  such  effects  are  expected  to  be  large.  A 
careful  preexperiment  analysis  will  help  determine  if  and  how 
this  can  best  be  accomplished.  Difficulties  of  this  type  can  be 
reduced  if  the  size  of  the  design  is  increased,  thereby 
increasing  the  number  of  available  trend-robust  columns. 

Additive  carryover  columns  correlate  with  trend  effects  in 
essentially  the  same  patterns  shown  by  the  direct  columns. 
Interactive  carryover  columns  are  all  trivially  correlated  with 
trends. 

SELECTING  THE  FINAL  TCR  DESIGN 

Up  to  this  point,  we  have  tentatively  been  considering  the 
relationships  among  columns  with  new  factor  labels:  A,  B,  C,  D, 
F,  G,  H,  L,  and  M  to  be  used  for  direct  main  effects.  However, 
in  a  number  of  sections,  dangerous  and  unacceptable  correlations 
occurred  too  frequently  when  columns  L  and  M  were  part  of  the 
new  factor  label. 

That  fact,  along  with  severe  limitations  in  the  total 
available  degrees  of  freedom  in  the  N  »  32  design,  suggests  that 
columns  L  and  M  should  not  be  considered  main  effect  columns. 
This  would  leave  us  with  a  seven  factor.  Resolution  IV  design, 
with  the  experimental  conditions  being  defined  by  the  columns 
(new  factor  labels) :  A,  B,  C,  D,  F,  G,  and  H.  Each  of  these  - 
direct  main  effect  columns  are  all  ones  with  an  AB-term  in  its 
original  factor  label. 

The  original  factor  label  column  AB  itself  is  not  included 
as  a  main  effect  column;  it  is  a  new  factor  label  column  L. 

The  relevant  columns  for  the  direct  interaction-string  effects 
are:  AB+,  AC+,  AD+,  AF+,  AG+,  AH+,  and  AL+.  [8] 

This  design  is  viable  only  if  the  two  restrictions  cited 
earlier  are  applicable,  namely:  (1)  additive  carryover 
interaction  effects  are  not  critical,  and  (2)  the  design  is  not 
used  to  estimate  interactive  carryover  effects. 


The  pattern  of  nontrivial  correlations  for  this  seven 
factor  (N  =  32)  Resolution  IV,  trendi-carryover  robust  design  is 
provided  in  Tables  4-A,  and  4”C.  Letters  M,  D,  and  U  are 

used  to  indicate  the  nontrivial  levels  —  Marginal,  Dangerous, 
and  Unacceptable,  respectively.  An  empty  space  indicates  that 
the  correlation  is  either  trivial  or  zero.  The  percent  overlap 
for  each  marginal  correlation  follows.  An  investigator  would 
use  these  tables  to  decide  how  best  to  assign  experimental 
factors  to  the  columns. 


No  table  is  provided  for  the  intercorrelations  among  the 
direct  columns  and  among  the  interactive  carryover  columns  since 
both  are  diagonal  matrices,  i.e.,  with  ones  along  the  diagonal 
when  a  column  is  correlated  with  itself  and  zeroes  off  the 
diagonal. 

SUMMARY  TO  TCR  DESIGN  CONSTRUCTION  PROCEDURES 

The  following  steps  will  produce  a  2^“^  trendi-carryover 
robust  design  of  Resolution  IV: 


(1)  To  construct  a  design  capable  of  studying  k  factors, 
each  at  two  levels,  one  must  use  the  least  number  of  independent 
conditions  required  to  satisfy  this  equation: 

N  *  [2^  >  4(k+l)J 

where  k  equals  the  number  of  factors  to  be  studied  and  x  equals 
(k-p) .  The  reciprocal  of  2P  indicates  what  proportion  the  size 
of  this  design  is  to  the  full  2^  factorial. 

Example  1.  To  determine  the  size  of  a  design  needed  to 
study  seven  factors,  we  substitute  7  for  k  and  get  4(7+1)  =  32, 
That  makes  x  -  5  since  32  is  equal  to  two  to  the  fifth  power. 
With  k  =  7,  and  x  =  5,  then  5  =  (7-p)  and  p  equals  2.  Thus, 
this  design  would  be  of  a  size  l/2^or  one-fourth  of  the  full 
2 ^factorial.  This  is  true,  of  course,  since  the  design  requires 
32  conditions  and  a  2^  design  requires  128. 


Example  2.  To  determine  the  size  of  a  design  needed  to 
study  10  factors,  we  substitute  10  for  k  and  get  =  4(10+1)  =  44 
for  which  the  nearest  power  of  two  equal  to  or  greater  than  that 
number  is  2^  »  64.  With  k  =  10  and  x  =  6,  then  p  =  4.  The 
reciprocal  of  2^,  therefore,  indicates  that  the  design  is  a 
one-sixteenth  fraction  of  the  full  2-^^  factor ial , 

(2)  Construct  in  Yates'  standard  order  the  sign  matrix  for 
the  2®  factorial  with  the  value  of  x  being  determined  in  Step  1. 

(3)  Sort  on  A —  (original  labels) ,  That  is,  reorganize  the 
columns  as  described  earlier  and  illustrated  in  Table  1  to  build 
a  trend-robust  design.  At  this  point,  the  A^columns  would  be 
used  for  main  effects  and  the  non-A-columns  for  the  two-factor 
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TABLE  4.  CRITICAL  RELATIONSHIPS  AMONG  DIRECT,  CARRYOVER 
AND  TREND  EFFECT  COLUMNS  IN  A  SEVEN  FACTOR, 
QUARTER  REPLICATE,  RESOLUTION  IV  TREND-CARRYOVER 
ROBUST  SCREENING  DESIGN* 


NOTES s 

1.  Direct  vs  Direct  (Section  A  of  Table  2)  is  not  shown 
here  since  all  direct  colunms  are  orthogonal  to  one 
another. 

2.  Additive  Carryover  vs  Additive  Carryover  (Section  E  of 
Table  2)  is  not  shown  here  since  additive  carryover 
columns  are  all  orthogonal  to  one  another. 

3.  Interaction  Carryover  columns  in  Table  4-8  correlate 
the  same  (except  occasionally  for  the  sign)  with  both 
Direct  and  Additive  Carryover  columns. 

NOTATIONS ! 

Blank  space:  No  effect  or  trivial  correlations  (r  •  .00  to  .31). 

M:  Marginal  correlations  (r  ••  .32  to  .54).  Numljer  with  M 
is  the  percent  overlap,  i.e.,  r  squared,  of  the  two 
variables. 

0:  Dangerous  correlations  (r  ~  .55  to  .70). 

U:  Unacceptable  correlations  (r  >>.71). 

ASSUMPTIONS  TO  BE  MET: 

When  this  design  is  used  for  screening  purposes, 

1.  Direct  interaction  strings  have  no  critical  additive 
carryover  effects. 

2.  Interactive  carryover  effects  will  not  be  estimated. 


*A11  labels  in  Tables  4-A,  4-B,  and  4-C  are  "new  factor  labels" 


ADDITIVE  CARRYOVER 
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interaction  strings.  As  in  any  fractional  factorial,  all 
odd-order  effects  are  confounded  with  main  effects  and  all 
even-order  effects  are  confounded  with  two-factor  interaction 
strings.  Any  unravelling  required  would  follow  the  procedures 
used  for  any  screening  study  (Simon,  1977a) . 

(4)  To  make  a  design  also  carryover  robust,  sort  the  A-^- 
columns  into  AB—  and  non-AB—  groups,  with  the  exception  of 
column  AB  itself,  which  is  to  remain  with  the  non-AB —  columns. 
The  columns  in  the  AB —  group  can  be  used  as  main  effect  columns 
in  the  trend-carryover  robust  design. 


(5)  Identify  the  relevant  two-factor  interaction-string 
columns  by  finding  those  in  which  new  factor  label  A  interacts 
with  the  labels  used  as  main  effects,  as  well  as  all  other 
columns  in  which  the  generalized  interactions  among  main  effects 
appear.  (Column  aliases  will  have  to  be  worked  out  or  derived 
from  existing  tables;  see  Simon,  1977a). 
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(6)  All  unused  AB —  and  A —  columns  may  be  used  to  estimate 
three-factor  interaction  effects.  All  unused  non-A —  columns 
may  be  used  to  estimate  four-factor  interaction  effects. 

However,  as  will  be  discussed  in  the  next  section,  we  may  wish 
to  use  the  degrees  of  freedom  from  these  unused  columns  to 
partial  out  trend  and  carryover  effects,  as  well  as  to  provide 
an  estimate  of  internal  error  variance. 

Just  how  trend  and  carryover  robust  this  N  =  32,  Resolution 
IV  TCR  design  is  is  summarized  in  Table  5.  This  table  shows  the 
type  and  frequency  of  nontrivial  correlations  among  the  direct 
columns  and  all  other  columns.  The  number  in  parenthesis 
associated  with  each  cell  in  the  table  indicates  the  total 
number  of  correlations  considered  in  that  cell.  The  size  of 
nontrivial  correlations  among  columns  are  indicated  by  the  M,  D, 
U  coding  used  earlier.  Correlations  with  onesself  and 
correlations  between  direct  and  additive  carryover  interactions 
(which  our  working  hypothesis  has  excluded)  are  not  counted  in 
this  table.  Out  of  the  total  700  correlations  considered  in  the 
table,  17  (2.4%)  are  marginal,  3  (0.4%)  are  dangerous  or 
unacceptable,  and  97.2%  are  trivial  or  absent. 
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TABLE  5.  NUMBER  OF  NONTRIVIAL  CORRELATIONS  BETWEEN 
DIRECT  EFFECTS  AND  ALL  SOURCES  OF  VARIANCE  IN  THIS 
SEVEN  FACTOR  TCR  RESOLUTION  IV  DESIGN 

Direct  Additive  Interactive  Trend 


MAIN 

INT 

MAIN 

INT 

MAIN 

INT 

L 

Q 

C 

Main  Direct 

0 

0 

3-M 

3-M 

0 

2-M 

0 

0 

0 

(N) 

(42) 

(49) 

(49) 

(49) 

(49) 

(49) 

(21) 

(21) 

(21) 

Interaction 

Direct 

0 

0 

3-M 

0  * 

0 

2-M 

1-M 

1-M 

2-M 

(N) 

(49) 

(42) 

(49) 

(49) 

(49) 

(49) 

1-U 

(21) 

1*^U 

(21) 

1-D 

(21) 

*  Correlations  between  direct  and  additive  carryover  interaction 
columns  are  not  included,  as  per  the  limitation  stated  earlier. 


Since  the  design  is  not  completely  orthogonal  to  sequence 
effects  and  is  a  Resolution  IV  design  with  two-factor 
interactions  in  strings,  it  will  be  necessary  for  the 
investigator  to  take  some  reasonable  steps  to  optimize  its 
effectiveness.  First,  through  preexperiment  analysis,  he  must 
decide  whether  the  strengths  and  weaknesses  of  the  TCR  design 
fit  the  characteristics  of  his  particular  task.  This  frequently 
is  not  a  black  or  white  decision  and  involves  compromise  among  a 
number  of  criteria.  It  can  be  done  best  by  comparing  the 
information,  distortion,  and  costs  associated  with  this  design 
against  those  of  alternative  data  collection  plans.  How 
effectively  the  design  will  be  used  also  depends  on  the 
investigator's  skill  in  assigning  the  experimental  factors  to 
the  columns.  Preexperiment  analysis  combined  with  the 
information  in  Table  4  and  the  tables  in  Appendix  C  can  greatly 
facilitate  this.  It  must  be  remembered  that  assigning  a  factor 
known  to  have  a  small  or  no  carryover  effect  to  a  highly 
confounded  column,  and  vice  versa,  tends  to  neutralize 
considerably  the  weaknesses  from  the  lack  of  orthogonality. 
Finally,  the  design  will  be  most  effective  if  the  investigator 
employs  the  appropriate  analysis  (to  be  discussed  in  the  next 
section),  properly  interprets  the  results,  and  uses  them  to 
iterate  the  process  and  supplement  the  data,  if  necessary. 

In  an  actual  experiment,  following  the  screening  phase,  it 
is  likely  that  the  above  design  would  be  amended  by  collecting 
additional  data  as  prescribed  by  the  holistic  approach  (Simon, 
1977a;  1979)  .  The  purpose  of  adding  more  data  might  be  to  add 

new  factors,  shift  the  range  of  old  ones,  obtain  more  precise 


information  on  the  more  important  ones,  isolate  potentially 
critical  effects  from  their  aliased  sources,  and/or  improve  the 
fit  of  the  response  surface  (if  one  exists). 
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SECTION  VI 


ANALYSES  AND  INTERPRETATIONS 


Theoretically,  N-1  effects  can  be  isolated  from  a  design 
composed  of  N  independent  experimental  conditions.  In  the  case 
of  the  N  =  32,  TCR  design,  31  effects  can  be  isolated.  Because 
the  design  is  in  fact  oversaturated  with  identifiable  sources  of 
variance  (i.e.,  there  are  more  known  sources  of  variance  than 
there  are  degrees  of  freedom)  the  experimenter  must  decide  in 
which  way  he  will  cut  the  pie,  which  sources  he  will  try  to 
isolate.  As  in  any  oversaturated  design,  the  remaining  sources 
are  confounded  with  those  being  isolated,  introducing  the 
specter  of  possible  bias  should  they  prove  to  have  significant 
effects  of  their  own.  In  the  previous  sections,  a  TCR  design 
was  selected  that  would  reduce  the  risk  of  this  danger.  Still, 
the  experimenter  has  a  number  of  options  when  he  analyzes  the 
data  from  this  design  and  must  select  the  appropriate  one(s) 
taking  into  consideration  the  information  demanded  by  the 
experiment,  the  capability  of  the  design  to  provide  it, 
characteristics  of  the  real  world  that  provide  a  priori  guesses 
as  to  what  might  be  important,  and  the  cost  of  "drawing  wrong 
conclusions.  If  the  design  is  primarily  used  as  a  screening 
design,  precise  estimates  of  the  effects  are  less  important  at 
this  stage,  and  ordinarily  the  investigator  may  collect 
additional  data  to  resolve  some  ambiguities.  Furthermore,  the 
investigator  is  not  limited  to  a  single  analysis,  and  in  fact, 
if  the  term  "research"  is  to  be  taken  seriously,  he  will  "play" 
with  his  data  as  a  Daniel  or  a  Tukey  might  do.  Still,  a  word  of 
caution  is  in  order.  In  general,  selective  examination  of 
nonorthogonal  data  can  be  dangerous.  This  general  principle 
applies  here  and  the  investigator  should  be  cautious  in  his 
interpretation. 

Below,  some  alternative  analyses  are  suggested  for  the  N  = 
32,  TCR  Resolution  IV  design  just  developed.  The  investigator 
can  probably  think  of  others.  The  direct  effect  labels  apply  to 
the  columns  in  Table  1. 

ANALYSIS  1 

The  31  degrees  of  freedom  might  be  partitioned  in  the 
following  manner; 

7  MAIN  EFFECTS; 

Original  factor  labels;  All  with  an  AB —  term 

in  label  except  AB 

New  factor  labels  A,  B,  C,  D,  F,  G,  H 
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7  TWO-FACTOR  INTERACTION  STRINGS; 


Original  factor  labels  All  without  a  B— 

term  in  label 

New  factor  labels  AB+,  AC+,  AD+,  AF+,  AG+, 

AH+,  AL+ 

9  THREE-FACTOR  INTERACTION  STRINGS: 

Original  factor  labels  Unused  columns  with  an  A — 

term  in  label 

New  factor  labels  E,  I,  J,  K,  L,  M,  N,  0,  P 

8  FOUR-FACTOR  INTERACTION  STRINGS: 

Original  factor  labels  .  Remaining  columns,  all  with 

a  B —  term  in  label 

New  factor  labels  AE+,  AI+,  AJ+,  AK+,  AM+, 

AN+,  A0+,  AP+ 


In  this  partitioning,  no  error  variance  is  estimated  with 
which  to  do  a  conventional  test  of  statistical  significance. 

This  in  itself  is  not  a  serious  deficiency  since  it  is 
preferable  to  use  a  normal-order  plot  (Simon,  1977a;  Daniel, 
1976)  to  estimate  the  criticalness  of  the  effects  in  a 
2 ^~P  multi  factor  design  of  this  size.  Should  an  external 
estimate  of  error  still  be  desired  —  to  use  and  compare  with 
the  one  derived  from  the  normal-order  plot  as  well  as  to  use  as 
a  test  of  the  goodness  of  fit  of  the  data  —  then  data  might  be 
collected  at  the  center  of  the  experimental  space  (Simon,  1977a) 
if  the  factors  are  quantitative  and  continuous  and  have  a  center 
point.  Some  investigators  have  been  satisfied  with  an  estimate 
of  error  variance  obtained  by  combining  the  data  on  the 
four-factor  interaction  strings  based  on  the  assumption  that  it 
is  highly  unlikely  that  four-factor  interaction  effects  are  more 
than  a  chance  occurence. 

This  analysis  plan  follows  the  traditional  approach  which 
fails  to  recognize  the  existence  of  sequence  effects  during  the 
analysis  of  a  within-subj ect  design.  It  differs,  however,  to 
the  extent  that  the  TCR  design  is  likely  to  be  less  confounded 
with  trend  and  carryover  effects  than  most  available  fractional 
factorial  designs.  Still,  failing  to  remove  some  of  the 
marginally  confounded  carryover  and  trend  effects  may  create  a 
bias  and  is  risky,  considering  it  is  not  necessary.  One  might 
wish  to  contrast  the  results  from  this  analysis  with  those  from 
the  other  analyses  presented  here  insofar  as  the  direct  effects 
are  concerned.  Do  the  estimates  of  the  direct  effects  change 
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markedly  when  sequence  effects  are  removed?  Do  the  changes 
if  they  exist  —  crucially  affect  the  interpretation  of  the 
results  for  screening  purposes? 

ANALYSIS  2 

We  may  be  content  only  to  remove  the  trend  effects 
statistically  and  leave  whatever  carryover  effects  that  might  be 
present  confounded  with  the  direct  effects  (see  Table  4).  We 
may  partial  trends  from  the  experimental  effects  using  the 
equations  suggested  by  Daniel  and  Wilcoxon  (1966)  to  handle 
linear  and  quadratic  trends  and  supplemented  by  Simon  (1977a,  p. 
121-128)  to  handle  cubic  trends.  Since  the  removal  of  three 
trend  effects  require  three  additional  degrees  of  freedom,  we 
must  sacrifice  three  of  the  31  columns  —  those  which  are  least 
likely  to  be  statistically  significant  —  and  use  them  to 
partial  out  the  trends.  The  column  selected  must  correlate  with 
the  trend  effect  it  represents. 

Since  four-factor  interactions  are  least  likely  to  be 
critical,  then  we  might  select  columns  D,  E,  and  DE  (original 
labels)  to  represent  linear,  cubic,  and  quadratic  trends, 
respectively. 

ANALYSIS  3 

If  one  wished  to  estimate  the  effects  of  all  direct'  (main 
and  two-factor  interaction  strings) ,  carryover  (additive  and 
interactive),  and  trend  effects  in  this  N  =  32,  TCR  design,  45 
degrees  of  freedom  would  be  the  minimum  number  required.  Since 
only  31  degrees  of  freedom  are  available,  some  kind  of 
compromise  must  be  made.  The  goal  of  such  a  compromise  would  be 
to  estimate  the  31  effects  most  likely  to  be  significant  and 
ignore,  i.e.,  allow  to  remain  confounded,  effects  believed  on 
some  a  priori  basis  unlikely  to  be  critical. 

This  plan  allows  us  to  estimate  direct  effects,  both  main 
and  two-factor  interaction  strings,  and  also  additive  carryover 
main  effects.  This  might  be  useful  where  there  is  outside 
empirical  evidence  that  interactive  carryover  effects  are 
negligible. 

The  following  (not  totally  independent)  sources  of  variance 
would  be  selected  to  be  terms  in  a  regression  analysis: 

7  MAIN^DIRECT  EFFECTS; 

A,  B,  C,  D,  F,  G,  H 

7  TWO-FACTOR  INTERACTION  DIRECT  STRINGS; 

AB+,  AC+,  AD+,  AF+,  AG+ ,  AH+ ,  AL+ 

7  MAIN-ADDITIVE  CARRYOVER  EFFECTS; 

A' ,  B* ,  C ,  D' ,  F' ,  G' ,  H' 
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9  THREE-FACTOR  INTERACTION  STRINGS; 

E,  1,  J,  K,  L,  M,  N,  0,  P 


1  FOUR-FACTOR  INTERACTION  STRING: 

AE+ 

If  this  analysis  is  used,  the  following  conditions  must  be 
valid: 

1.  Interactive  carryover  effects  are  negligible. 

2.  Interactions  among  additive  carryover  effects 
are  negligible. 

3.  Direct  and  carryover  effects  do  not  interact 
critically  with  one  another. 

4.  Higher-order  interactions  {not  included  in  the 
strings  listed  above)  are  negligible. 

Whatever  degrees  of  freedom  are  available  from  noncritical 
three-factor  interaction  strings  can  be  used  to  isolate  trend 
effects  or  serve  as  an  internal  estimate  of  error  variance. 

Some  three-factor  interaction  strings  are  confounded  with  main 
effects.  If  from  the  preexperiment  analysis  the  investigator 
suspects  the  possibility  of  a  critical  three-factor  interaction, 
care  should  be  taken  to  assign  the  factors  so  that  the 
interaction  will  be  located  in  a  string  that  is  isolated  from 
main  effects.  While  main  effects  will  be  robust  against  trend 
and  interactive  carryover,  failure  to  isolate  any  component  of 
these  sequential  effects  that  is  critical  will  inflate  the  error 
variance  and  reduce  the  sensitivity  of  a  significance  test. 
Column  AE+  might  be  selected  from  among  the  unused  columns 
representing  four-factor  interaction  strings  to  isolate  the 
linear  and  cubic  trends. 

ANALYSIS  4 

An  analysis  might  be  performed  with  the  same  breakdown  as 
shown  above,  except  that  interactive  carryover  columns  are  used 
instead  of  the  additive  ones.  This  might  be  the  case  when  there 
is  some  evidence  that  they  are  likely  to  have  the  more  severe 
effects . 

However,  because  of  the  high  degree  of  intercorrelation 
among  interactive  carryover  effects,  the  analysis  of  their 
individual  terms  should  not  be  taken  seriously.  Instead,  their 
composite  variance  should  be  evaluated  for  statistical 
significance  to  decide  whether  further  isolation  is  needed. 
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ANALYSES  5A  AND  5B 


One  may  be  highly  selective  regarding  the  terms  (columns) 
that  will  be  included  in  an  analyses  in  a  way  that  will  maximize 
the  isolation  of  direct  effects  and  sacrifice  the  information 
regarding  carryover  effects  per  se.  Information  regarding 
carryover  would  be  used  solely  to  partial  these  effects  from  the 
direct  main  and  interaction  effects.  When  the  TCR  design  is 
used  for  screening  purposes,  this  is  probably  the  soundest 
analysis  of  them  all. 

Inspection  of  Table  4  shows  the  sources  of  variance  most 
likely  to  bias  direct  main  and  interaction  effects  in  the  N  = 

32,  TCR  design.  This  information  is  summarized  here  (new  factor 
labels) : 


Direct  Effects  Confounded  Sources 

A  A' ,  AL+’ 

B 

C  C,  AG+' 

D 

F  F* ,  AD+' ,  AC+",  AH+" 

G 

H 


AB+ 

AB+'  , 

LT, 

CT 

AC+ 

AC+’  , 

LT, 

CT 

AD+ 

AD+'  , 

F'  , 

AC+ 

AF+ 

AF+'  , 

QT 

AG+ 

AG+'  , 

c , 

QT 

AH+ 

AH+' 

AL+ 

AL+'  , 

A'  , 

CT 

The  sources  to  serve  as  terms  in  a  regression  analysis  are: 

7  Direct  main  effects:  A,  B,  C,  D,  F,  G,  H 

7  Direct  interaction  effects:  AB+,  AC+,  AD+, 

AF+,  AG+,  AH+,  AL+ 

10  Additive  carryover  effects:  A*,  C',  F' ,  AB+',  AC+' , 

AD+'  ,  AF+'  ,  AG+'  ,  AH+'  ,  AL+' 

2  Interactive  carryover  effects:  AC+",  AH+" 

3  Trend  effects;  LT,  QT,  CT 


2  Residual 


This  plan  uses  29  of  the  available  31  degrees  of  freedom 
for  identifiable  effects,  leaving  two  for  whatever  purpose  the 
experimenter  deems  important  to  isolate. 

Given  that  plan,  the  investigator  may  approach  the  analysis 
in  several  ways; 

A.  DEVELOP  A  PREDICTION  EQUATION.  Prepare  a  matrix 
composed  only  of  the  columns  for  the  above  terms  along  with  the 
performance  associated  with  each  experimental  condition. 

Perform  a  multiple  (possibly  stepwise)  regression  analysis. 

B.  ESTIMATE  DIRECT  EFFECTS.  Partial  from  each  direct 
effect  the  source(s)  specifically  confounded  with  it  to  obtain  a 
less  confounded  estimate  of  the  particular  effect  (see  Daniel, 
1976,  p.  226-232,  for  ways  of  examining  partially  confounded 
data)  . 


The  exact  strategies  for  these  analyses  are  not  supplied 
since  circumstances  will  differ  and  the  lines  to  follow  are  best 
left  up  to  the  ingenuity  of  the  investigator.  There  may  be 
other  more  appropriate  analyses  for  particular  purposes.  Those 
readers  who  have  used  a  "cookbook"  approach  to  experimentation 
in  the  past  —  canned  designs  from  a  book  and  the  results  from  a 
computerized  analysis  of  variance  —  may  be  uncomfortable  with 
the  task  of  having  to  think  about  the  analysis  and 
interpretation  and  to  have  to  work  with  effects  that  arc  only 
quasi-or thogonal ,  If  so,  as  a  general  model,  one  may  find 
Daniel's  book  (1976)  a  breath  of  fresh  air  in  this  regard,  as  he 
illustrates  the  thought  processes  employed  to  extract  useful 
information  from  slightly  marred  data.  Where  limitations  on 
subjects  and  data  collection  time  permit,  a  larger  design  might 
be  employed  to  provide  more  trend-carryover  robust  columns  from 
which  to  select.  Here  th,e  investigator  must  decide  whether  that 
approach  is  more  cost-effective  than  accepting  some  uncertainty 
in  the  first  stage  of  the  screening  process  and  using  their 
resources  in  a  second  stage  once  the  location  of  ambiguities 
have  been  bt . cer  identified. 

For  those  who  question  the  value  of  a  design  that  requires 
such  exercises,  were  they  not  to  use  a  TCR  design  in  the 
presence  of  possible  sequence  effects,  then  they  must  suggest  an 
alternative  plan  that  can  examine  (screen)  a  great  many  factors 
as  economically  while  keeping  their  estimated  effects  robust 
against  the  possible  biases.  Certainly  ignoring  or  overlooking 
the  problem,  the  historical  "solution"  for  the  past  100  years, 
is  not  the  answer. 
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SECTION  VII 


CONCLUSIONS 


When  a  single  subject  is  tested  serially  on  all  conditions 
of  an  experiment,  a  2^pP  holistic  design  can  be  modified  in  a 
way  that  will  make  the^ experimental  effects  robust  against 
linear,  quadratic,  and  cubic  trends  as  well  as  additive  and 
interactive  carryover  effects  that  may  exist.  Robustness 
relates  more  to  the  effect  —  the  difference  between  means  — 
than  to  the  means  themselves.  This  makes  this  design 
particularly  useful  as  a  screening  device  with  which  the  more 
important  factors  out  of  a  large  number  of  potentially  critical 
ones  can  be  identified.  Valid  mean  values  are  better  obtained 
after  the  number  of  conditions  have  been  markedly  reduced  to  a 
relatively  few,  selected  by  the  screening  process.  At  that 
time,  each  conf iguration  would  be  evaluated  under  operational 
conditions  using  different  groups  of  representative  subjects. 

N  =  64  TCR  designs  should  be  particularly  attractive  since 
they  not  only  would  allow  10  to  12  factors  to  be  examined  using 
a  single  subject  with  minimum  concern  for  sequence  effects,  but 
the  size  is  enough  to  provide  reasonably  precise  and 
generalizable  answers  at  marginal  costs.  Additional  data  might 
be  required  once  the  initial  block  has  been  collected  in 
accordance  with  the  principles  associated  with  the  holistic 
approach  and  screening  studies  (Simon,  1977a,  1977b,  1979). 

The  design  presented  in  this  report  demonstrates  that  it  is 
possible  to  have  a  practical,  cost-'ef f ective  TCR  2  holistic 
design.  If  one  wished  to  estimate  both  direct  and  "^(^rryover 
effects,  then  totally  new  designs  for  that  purpose  must  be 
developed  instead  of  modifying  existing  ones  as  was  done  in  this 
report.  While  such  designs  are  likely  to  require  that  more  data 
be  collected,  the  additional  information  gained  would  ordinarily 
justify  the  cost. 


FOOTNOTES 


[1]  A  holistic  approach  to  behavioral  research  which  emphasizes  the 

importance  of  accounting  for  as  many  potentially  critical 
variables  as  possible,  whether  equipment,  environment,  subject, 
or  temporal,  controlled  or  uncontrolled,  whether  generated  in 
the  real  world  for  a  specific  task  or  within  the  experimental 
setting.  This  approach  involves  a  sequential  strategy  and 
bundle  of  techniques  that  enables  the  response  surface  of  a 
large  multidimensional  space  ot  practical  importance  to  be 
estimated  while  reducing  irrelevant  sources  of  variance,  all  at 
minimum  data  collection  costs. 

[2]  When  factors  are  aliased  in  2^  or  2^~P  designs,  the  experimental 

conditions  being  compared  to  determine  each  factor's  effects  are 
identical,  except  for  a  possible  reversal  of  signs.  When  it  has 
been  determined  that  the  effect  of  a  string  composed  of  aliased 
two-factor  interactions  is  significantly  large,  it  is  not 
possible  to  determine,  without  additional  data  collection,  which 
interactions  in  the  string  are  responsible  for  the  observed 
effect.  For  example,  the  column  labelled  AB+  in  Table  1  would 
actually  be  measuring  the  combined  effects  of  a  string  of 
two-factor  interactions,  specifically,  the  combined  effects  of 
(AB  +  CE  +  DF  +  GH) ,  with  each  term  contributing  an  unknown 
amount  (Appendix  II) .  Fractional  factorial  designs  of  the  same 
size  and  resolution  can  differ  as  to  the  number  of  two-factor 
interactions  that  will  be  confounded  (Fries  and  Hunter,  1978). 
The  experimenter  must  decide  which  designs  best  suits  his  needs. 

[3]  Some  published  fractional  factorial  designs  (e.g.,  by  the  U.S. 

Department  of  Commerce,  1957)  may  be  trend  robust  and  have  a 
better  aberration  pattern  than  the  holistic  design  described 
here.  However,  some  may  not  be  trend  robust  and  there  is 
usually  no  indication  either  way  to  help  the  user  decide.  They 
may  be  used,  of  course,  provided  the  experimenter  first 
identifies  what  the  original  factor  labels  of  the  columns  are 
(using  the  letters  and  patterns  of  Yates'  standard  order  and  by 
referencing  Table  1)  and  does  not  use  those  too  seriously 
confounded  with  trend  effects. 

[4]  No  taxonomy  has  ever  been  developed  that  relates  tasks  or  task 

characteristics  to  the  types  and  strengths  of  carryover  and 
other  sequence  effects  that  are  likely  to  exist.  That  would  be 
a  worthy  project  and  one  of  considerable  value  for  optimizing 
the  selection  and  effective  use  of  an  experimental  design. 
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[5]  This  sequence  is  the  AB  sign  pattern  in  a  conventional 

2*  factorial  design  arranged  in  Yates'  standard  order. 

Repeating  the  ABBA  pattern  for  any  within-subject  plan  of  length 
N  =  2*  is  the  best  alternative  for  sequencing  two  conditions 
when  trends  and  additive  and  interactive  carryover  effects  are 
considered  potentially  critical.  As  in  the  case  when  any 
experimental  design  is  selected,  the  investigator  must  determine 
whether  the  choice  is  suitable  for  the  particular  task  being 
studied.  For  example,  the  design  would  not  be  appropriate  if 
the  carryover  from  any  early  condition  resulted  in  a  permanent 
change  throughout  the  remainder  of  the  experiment  since  the  plan 
is  to  be  used  only  when  the  effects  last  for  essentially  one 
trial.  Designs  for  examining  more  than  one  trial  carryover 
effect  do  exist  (Simon,  1974)  . 

[6]  Actually,  the  sources  of  variance  are  not  fully  isolatable. 

While  the  direct  and  carryover  patterns  for  the  ABBA —  pattern 
are  orthogonal,  intercorrelations  between  direct  or  carryover 
patterns  and  linear,  quadratic,  or  cubic  trends  are  not  totally 
independent.  There  are  small  correlations  which  for  all 
practical  purposes  are  inconsequential  and  do  not  destroy  the 
effectiveness  of  the  data  collection  plan.  Specifically,  the 
direct  pattern  correlates  0.0131  with  quadratic  trend;  this  is 
a  0.02%  overlap.  The  additive  carryover  pattern  for  this 
sequence  correlates  -0.1083  with  linear  trend  and  -0.1634  with 
cubic  trend;  these  are  1.17%  and  2.67%  overlaps  respectively. 
The  interactive  carryover  pattern  for  this  sequence  correlates 
-0.0542  with  linear  trend  and  -0.0829  with  cubic  trend;  these 
are  0.29%  and  0.69%  overlaps  respectively. 

[7]  When  the  experimental  conditions  of  a  2^  factorial  design  are 

arranged  in  Yates'  standard  order,  the  sign  patterns  for  the 
columns  of  a  2 ^  factorial  design,  for  example,  are  the  same  as 
those  for  any  2^  condition,  two-level  fractional  factorial 
design.  For  example,  the  sign  patterns  of  the  columns  of  a 
2®  factorial  are  the  same  as  those  for  a  ,  or 

22S-8  fractional  factorial  design.  The  correlations  among  the 
columns  are  the  same  for  all  two-level  factorial-like  designs 
with  32  conditions.  Rearranging  the  columns  of  one  of  these 
designs  to  form  the  trend-robust  holistic  design  does  not  affect' 
the  correlation  once  the  proper  labels  are  identified. 

[8]  Note  that  AL+  is  the  only  one  of  the  interaction-string  columns 

in  which  both  letters  in  its  label  are  not  found  represented 
among  the  main  effect  labels;  L  is  not  there.  Actually,  the 
visible  letters  of  a  string's  label  are  only  the  name  of  the 
string,  not  necessarily  the  interactions  of  interest  within  the 
string.  In  the  present  design,  adapted  from  that  shown  in  Table 
1,  the  21  relevant  two-factor  interactions  are  aliased  three  to 
a  column,  including  the  column  designated  new  label  AL-i-  (see 
Appendix  B) . 


NOTATIONS* 


2^~P  design 

A  fractional  factorial  design  capable  of  isolating  the 
effects  of  K  factors  at  two  levels  each  using  a  1/2^  fraction  of 
the  total  design.  If  p  equal  0,  the  design  is  a  full  factorial. 
If  a  subscript,  IV,  is  added,  that  indicates  that  the  design  is 
of  Resolution  IV. 

A,  A* ,  A* ,  and  other  letters  from  A  through  P 

Used  to  label  the  columns  of  the  experimental  design.  No 
superscript  indicates  a  column  that  represents  a  direct  effect; 
a  single  one  (')#  an  additive  carryover  effect;  and  a  double 
one  (") ,  an  interactive  carryover  effect. 


A — ,  AB — 


Represents  a  main  or  interaction  term  with  an  A  or  an  AB  in 
its  label.  For  example,  ACDE,  AF,  or  ABC,  ABDF,  ABE,  etc. 

AB-t,  AB+'  ,  AB-f" ,  and  other  letters  from  A  through  P 

The  plus  sign  indicates  that  the  designated  interaction  is 
the  label  of  a  column  containing  a  string  of  interactions. 

D 


When  referring  to  a  correlation,  indicates  that  its 
magnitude  for  a  TCR  design  is  "dangerously"  high,  i.e.,  from  .55 
and  .70,  making  the  percent  overlap  equal  to  38%  to  49%.  Its 
symbol  in  Table  3  is  g  • 


*  Letters  A  to  P  are  used  as  new  factor  labels  for  main  effects 
in  the  experimental  design  discussed  in  this  paper.  Some 

of  these  letters  are  also  used  to  represent  other  things.  The 
distinction  should  be  clear  from  the  context. 


The  identity  column  of  a  2^  factorial  or  fractional 
factorial  design.  All  signs  are  plus.  -[I]  would  be  used  if 
all  signs  in  the  column  are  minus. 

M 

When  referring  to  a  correlation,  indicates  that  its 
magnitude  for  a  TCR  design  is  "moderately"  high,  i.e.,  from  .32 
to  .54,  making  the  percent  overlap  equal  to  10%  to  29%.  Its 
symbol  in  Table  3  is  g. 

T 


When  referring  to  a  correlation,  indicates  that  its 
magnitude  for  a  TCR  design  is  "trivial,"  i.e.,  from  0+  to  .31, 
making  the  percent  overlap  equal  to  .00+  to  .09.  Its  symbol  in 
Table  3  is  . 

□ 


TCR  design 


Design  that  is  robust  against  trends  and  carryover  effects. 


U 

When  referring  to  a  correlation,  indicates  that  its 
magnitude  for  TCR  design  is  "unacceptably"  high,  i.e.,  from  .71 
or  higher,  making  the  percent  overlap  equal  to  50%  or  higher. 
Its  symbol  in  Table  3  is  |||  . 


GLOSSARY 


aliases 


Term  used  when  sources  of 
design  are  100%  confounded  and 
independently  estimated.  This 
fractional  factorial  design,  a 
effect  also  estimates  one  or  mo 

be tween- subject  design 


variance  in  an  experimental 
their  effects  cannot  be 
aliasing  occurs  when,  in  a 
contrast  that  estimates  one 
re  other  effects. 


Design  in  which  a  different  subject  or  subjects  are  tested 
on  each  experimental  condition, 

carryover  effect 


When  the  characteristics  of  a  condition  affects  the 
performance  on  a  different  condition  that  follows  it,  carryover 
is  said  to  have  taken  place.  In  the  literature,  the  terms 
"carryover,"  "crossover,"  and  "transfer"  have  been  used 
synonymously.  Additive  carryover  occurs  when  the  size  of  the 
carryover  effect  is  the  same  for  any  condition  whatever  the 
condition  that  follows  it  might  be.  Interactive  carryover 
occurs  when  the  size  of  a  carryover  effect  for  a  condition 
differs  depending  on  what  condition  follows  it. 


holistic  approach 

A  unique  sequential  process  for  ultimately  mapping  a  large 
mul ti f actor-mul tivar iate  space.  The  approach  is  composed  of  a 
philosophy,  a  strategy,  and  a  bundle  of  techniques  that  require 
and  enable  an  investigator  to  economically  study  under 
controlled  conditions  "all"  of  the  relevant  factors  affecting 
performance  on  a  specific  task.  Factors  to  be  considered  are 
from  sources  associated  with  equipment,  environment,  subject,  o 
temporal  conditions,  as  well  as  unwanted  biases  that  must  be 
reduced . 


new  factor  labels 

The  labels  for  the  experimental  designs  used  when  the  sign 
matrix  is  perceived  as  a  fractional  factorial.  This  would  be 
the  case  when  it  is  used  as  a  screening  design. 


ri'i  if^ii? 


rvi  '.I'l  ijB'ij)  ll■l  n 


original  factor  labels 

The  labels  used  when  the  sign  matrix  is  perceived  as  a  full 
factorial  in  Yates'  standard  order.  These  labels  are  most 
useful  when  one  must  mentally  manipulate  the  design. 

percent  overlap 

This  refers  to  the  proportion  (times  100)  of  confounding 
between  two  columns  or  factors  in  an  experimental  design. 

Percent  overlap  equals  the  correlation  squared. 

Resolution  IV 

Term  used  to  indicate  the  extent  and  nature  of  the  aliasing 
in  a  fractional  factorial  design.  The  IV  indicate  that  all  main 
effects  are  independent  of  one  another  and  of  all  two-factor 
interactions,  and  the  two-factor  interactions  are  aliased  within 
independent  two-factor  interaction  strings. 

screening  design 

A  fractional  factorial  design  capable  of  studying  the 
effects  of  a  large  number  of  factors  economically  under 
controlled  conditions  in  order  to  discover  which  few  are 
primarily  responsible  for  the  performance  on  a  particular  task. 
It  is  used  most  effectively  when  employed  with  the  holistic 
approach. 

string ,  interaction 

A  group  of  totally  aliased  interactions,  the  effects  of 
which  cannot  be  independently  estimated.  In  this  report,  an 
interaction  string  is  designated  by  a  plus  sign  following  the 
interaction  label,  e.g.,  AB+, 

trend 

An  effect  that  progresses  systematically  from  trial  to 
trial  through  the  experiment.  Of  interest  in  this  report  are 
linear  (L  or  LT) ,  quadratic  (Q  or  QT) ,  and  cubic  (C  or  CT) 
trends. 

within-subject  design 

Design  in  which  a  subject  is  tested  serially  on  all  the 
conditions  of  an  experiment. 
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APPENDIX  A 


MAIN  AND  THREE-FACTOR  INTERACTION  EFFECTS  ALIASED 
IN  THE  2{S'“  TREND  RESISTANT  SCREENING  DESIGN  IN 
TABLE  1. 


NEW  ORIGINAL 

MAIN  FACTOR  ALIASED  THREE-FACTOR  INTERACTION  STRINGS 

EFFECTS  LABELS 


0 

AE 

ABP 

ACK 

ADO 

CFP 

CGI 

CHJ 

HLP 

UN 

IKM 

K 

ADE 

ABN 

ACO 

ADE 

CFN 

CGM 

CIL 

HLN 

UP 

I  MO 

N 

AD 

ABK 

ACP 

ADI 

CFK 

CGJ 

CHI 

HKL 

UO 

IMP 

0 

ACE 

ABM 

ACE 

ADO 

CGN 

CHO 

CLP 

HLM 

IKP 

I  NO 

F 

ABC 

ABC 

AOL 

AEM 

CHL 

COM 

CKN 

HJP 

HMO 

ILN 

L 

AB 

A8H 

ACG 

ADF 

CIK 

CJP 

CMO 

GKO 

GNP 

HJM 

P 

A 

ABO 

ACN 

ADM 

CFO 

CHM 

CJL 

HLO 

UK 

IMN 

E 

ACDE 

ABI 

ACJ 

ADK 

CGP 

CHK 

CLN 

I  OP 

JKO 

JNP 

I 

ACD 

ABE 

ACM 

AON 

CGO 

CHN 

CKL 

GLM 

JKP 

JNO 

M 

AC 

A8J 

AC  I 

AOP 

CGK 

CHP 

CLO 

HJL 

IKO 

INP 

H 

ABE 

ABL 

ACO 

AEO 

CIN 

CJO 

CMP 

GKP 

GNO 

OLM 

D 

ABDE 

ABG 

ACH 

AEK 

CIP 

CJK 

CMN 

HKP 

HNO 

ILM 

C 

ABCE 

ABF 

AOH 

AEJ 

DIP 

OJK 

DMN 

HJO 

HMP 

IKL 

A 

ABCDE 

BCF 

GOG 

GEI 

DIN 

DJO 

OMP 

HJK 

HMN 

ILO 

B 

ABCD 

ACF 

AD6 

AEI 

DIK 

DJP 

DM0 

HJN 

HKM 

ILP 

G 

ABO 

ABD 

ACL 

AEN 

CIO 

CJN 

CKM 

HKP 

HNO 

ILM 

AEH  AFN  AGM  AIL  BCN  BDM 
CLM  OF  I  OGP  OHK  DLN  EF6 

JMP  KNP 

AFP  AGI  AHJ  ALM  BCP  BDI 

OFM  OGN  DHO  DLP  EFL  EIN 

JMN  NOP 

AEG  AFO  AKM  AJL  BCO  BDE 

OFJ  OGK  DHP  DLO  EFH  ElK 

JKM  KOP 

AFI  AGP  AHK  ALN  BCI  BDP 

DEH  DFN  DGM  DIL  EGL  EIM 

KMN  MOP 

AGH  AIJ  AKP  ANO  BDH  BEJ 

COP  DEP  DIO  DON  DKM  EGO 

JLO  LMP 

AEP  AIO  AJN  AKM  BCD  BEO 

DEM  DGH  OIJ  DKP  DNO  EFK 

HKN  HOP 

AEL  AFK  AGO  AHI  BCK  BDJ 

OEF  OGO  DHN  DKL  EIO  EON 

JMO  KNO 

AFM  AGN  AHO  ALP  BCM  BDN 

OFP  DGI  OHJ  DLM  FGO  FHN 

KMP  MNO 

AFJ  AGK  AHP  ALO  BCJ  BDK 

DEG  OFO  DHM  DJL  EHL  EJM 

KMO  MNP 

AEF  AGO  AHN  AKL  BCE  BOO 

DEL  DFK  OGJ  DHI  E6H  EIJ 

OKN  JOP 

AFG  AIP  AJK  AMN  BCG  BDF 

OEJ  DGL  DIM  DKO  DNP  EFN 

KLN  LOP 


AFL  AIN  AJO  AMP  BCL  BEN 

EFP  EGI  EHJ  ELM  FIO  FJN 

KLO  LNP 

AGL  AIM  AKO  ANP  BDL  BEM 

EFI  EGP  EHK  ELN  FHL  FJM 

JLP  LMO 

6HL  BJN  BKN  GOP  CDH  CEJ 

EFM  EGN  EHO  ELP  FGH  FIJ 

JLN  KLM 

AHL  AJM  AKN  AOP  CDL  CEM 

EFJ  EGK  EHP  ELO  FGL  FIM 

JKL  LMN 

AFH  AIK  AIP  AMO  BCH  BEK 

DEI  DHL  DJM  DKN  OOP  EFO 

KLO  LNP 


BEL  BFK  BGJ  BHI  CDE  GKL 

EIP  EJK  EMN  FHM  FJL  GHN 


BEG  BFO  BHM  BJL  CDJ  CEH 

EJO  EMP  F6J  FHI  GHP  GLO 


BFP  BGI  BHJ  BIM  COM  CEL 

EJP  EMO  FGM  FIL  GHO  GLP 


BEF  BGO  BHN  BKL  COK  CFM 
EKO  ENP  FGK  FHP  FLO  GHI 


BGL  BIM  BKO  BNP  CDG  CEI 

EHN  EKL  GIP  GJK  GMN  HIK 


BFG  BIP  BJK  BMN  CEN  CFH 

EGJ  EHI  fin  FJO  FMP  GIM 


BEH  BFN  BGM  BIL  CDI  CEG 
EKM  FGI  FHJ  FLH  GHK  GLN 


BFJ  BGK  BHP  BLO  CDO  CFI 

FKL  6HM  GJL  HIL  IJM  IKN 


BFM  BGN  BHO  BLP  COP  CEF 
EKN  EOP  FGP  FHK  FLN  GHJ 


BFI  BGP  BHK  BLN  CON  CFJ 

EKP  ENO  FGN  FHO  FLP  GIL 


BEP  BIO  8JN  BKM  CEK  CFL 

EGM  EIL  FIK  FJP  FMO  6IJ 


BFH  BIK  BJP  BHO  CEO  CFG 

FKN  GHL  GJH  6KN  GOP  HIM  - 


BGH  BIvI  BKP  BNO  DEO  DFG 
FKN  FOP  GIO  GJN  GKM  HIN 


CGL  CIM  CKO  CNP  OEK  OFL 

FKP  FNO  GIK  GJP  GMO  HIP 


CGH  CIO  CKP  CNO  DEN  DFH 

FKO  FNP  GIN  CJO  GHP  HIO 


BFI  BIN  BJO  BMP  CDF  CEP 

EHM  EJL  FIP  FJK  FMN  HU 


55 


M 


&3)-(i^«SZOO< 

m 

i<03OQ(i4U®»-3 

u 

«aOW04Hh>S 

03 

f<03UQfae>SS 

Q 

u 

oa4«>^®aaiJ 

03 

<03UQMF^C>H 

M 

a 

u 

03 

PjQasJaiDH 
.  <<fflUQW(nUa5 

u 

u 

a«03HOoai^ 

03 

rt:03OQwij42i-3 

w 

h3 

Q 

tDafaoH04«z 

U 

<9 

03 

<03UQpMUah3 

h3 

U 

Oi 

Q 

2  t->  H  (i<  [i,  o  a  t-3 

03 

•<03UQMe>ai< 

O 

Eh 

U 

< 

U 

Mwaat^^pno 

fu 

03 

rt:03UQfaO03h3 

a 

< 

a 

Qow«*JaoPi 

H 

o 

H 

on 

o 

U 

<03owf»4Hi-3a 

a 

a»4Qoo04u:a 

o 

^03UW&(Hi^g 

u 

Q 

j®ufeP<oaa 

U 

rti03OQWHiT>« 

U 

OQ^3Z«^>|^0^O 

U 

a;auu(tiHi^S 

Vba;>r>>-iSoat 

Q 

rt;03QWOH«Z 

U 

hUiJSa'-jCMO 

Q 

I^03QWOH«a 

0qCiHOHH3aZft 

U 

^uQwacjao 

'Cl 


U5  ^ 

iz  cn 

h  os  o 

O  Q  O  H  W 
W  Eh  E-I  03 

cn  cn  u  u 
o  <  <  <  iJ 
a  M  « 

H  I  w  5 
«  <  O  £H  W 

en  s  z  a 

W  Eh  M  '-' 


IWWi? 


APPENDIX  C 


INTERCORRELATIONS  AMONG  DIRECT,  CARRYOVER  AND  TREND  EFFECT  COLUMNS 
(2®  factorial  or  2  ^  ^ ^ fractional  factorial) 


A  complete  table  of  intercorrelations  for  a  five-factor, 
two-levels  per  factor,  factorial  with  direct,  additive 
carryover,  and  interactive  carryover  columns  would  involve  a  93 
X  93  matrix,  symmetrical  about  the  diagonal.  This  matrix  can  be 
conveniently  sectioned  into  nine  31  x  31  submatrices  as  shown  in 
Table  2.  However,  as  discussed  in  the  body  of  this  report, 
because  some  sections  are  duplicates  of  others  and  because  some 
are  unit  matrices,  it  is  only  necessary  to  show  three  sections 
to  allow  a  user  to  know  the  intercorrelations  among  all  of  the 
93  X  93  combinations  in  the  complete  matrix.  To  examine  the 
robustness  of  the  columns  against  linear,  quadratic,  and  cubic 
trend  effects,  three  additional  rows  of  correlations  are  added 
to  each  matrix,  making  it  a  34  x  31  intercorrelation  table. 

These  tables  —  the  numerical  equivalents  of  those  in  Table  3  — 
are  shown  in  this  appendix. 

Since  the  sign  patterns  in  the  columns  and  the 
relationships  among  the  columns  remain  the  same  for  equivalent¬ 
sized  fractional-factorial  designs,  the  information  provided 
here  applies  to  a  2^®’^^  fractional  factorial  design  and 
comparable  designs.  The  column  labels  are  changed  for  that 
purpose  and  aliasing  occurs.  These  relationships  are  discussed 
in  the  body  of  the  text  and  by  Simon  (1973;  1977a;  1981) . 

Employing  the  labels  and  notations  used  in  Tables  1,  3,  and 
4,  the  new  factor  label  equivalents  of  the  original  factor 
labels  found  in  this  appendix  are  as  follows: 


Labels  Labels  Labels  Labels 


Orig. 

New 

Orig. 

New 

Orig. 

New 

Orig. 

New 

A 

P 

D 

AC+ 

E 

AB+ 

DE 

AF+ 

B 

AE+ 

AD 

N 

AE 

0 

ADE 

K 

AB 

L 

BD 

AJ+ 

BE 

AI  + 

BDE 

AM+ 

C 

AD+ 

ABD 

G 

ABE 

H 

ABDE 

C 

AC 

M 

CD 

AH+ 

CE 

AG+ 

CDE 

AL+ 

BC 

AK+ 

ACD 

I 

ACE 

J 

ACDE 

E 

ABC 

F 

BCD 

A0+ 

BCE 

AN+ 

BCDE 

AP+ 

ABCD 

B 

ABCE 

C 

ABODE 

A 

The  plus  sign  after  each  new  label  two-factor  interaction 
indicates  that  it  is  the  label  of  a  string  of  aliased  two-factor 
interactions  (see  Appendix  B) .  Nothing,  an  apostrophe,  or  a 
quotation  mark  would  follow  each  label  to  indicate  whether  the 
labels  apply  to  direct,  additive  carryover,  or  interactive 
carryover  columns,  respectively. 
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NATBIX  C-1 


ADDITIVB  CARRYOVER 


A* 

B' 

AB* 

c* 

AC 

BC 

ABC 

O' 

« 

•1.0000 

0.0000 

0.0000 

0.0000 

0,0000 

0,0000- 

0.0000 

0.0000 

b 

0.0000 

0.0000 

•1.0000 

o.oooo 

0,0000 

o.cooo 

0,0000 

0.0000 

AH 

0.0000 

1 .0000 

0.0000 

0.0000 

O.Onoo 

0.0000 

O.OUOO 

0.0000 

C 

0.0000 

0.0000 

0.0000 

o.sooo 

•o.sooo 

•0,5000 

•o.sooo 

O.OUOO 

AC 

0.0000 

0.0000 

0.0000 

C.SOOO 

•o.sooo 

o.sooo 

o.sooo 

0,0000 

BC 

0.0000 

0.0000 

0.0000 

o.sooo 

o.sooo 

o.sooo 

•o.sooo 

0.0000 

ABC 

0.0000 

0.0000 

0.0000 

•o.sooo 

•o.sooo 

0.5000 

•o.sooo 

0.0000 

D 

0.0000 

,  0.0000 

o.oooo 

0.0000 

0.0000 

0.0000 

0.0000 

0.7S00 

AO 

'  0.0000 

0.0000 

o.oooo 

o.oooo 

0.0000 

0.0000 

0.0000 

0.2500 

BD 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.2500 

ABO 

0.0000 

o.oooo 

0.0000 

o.oooo 

o.oooo 

0.0000 

o.cooo 

-0.2SO0 

CO 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.2SOU 

ACO 

0.0000 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

•0.2500 

BCO 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

•0.2SUO 

ABCU 

o.oooo 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.2500 

e 

0.0000 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

AE 

0.0000 

0.0000 

0.0000 

o.oooo 

0.0000 

0.0000 

0.0000 

0.0000 

BE 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

0.0000 

0.0000 

O.OUOO 

ABE 

0.0000 

O.OoOO 

0.0000 

0.0000 

0.0000 

0.0000. 

o.oooo 

o.oooo 

Cf 

0.0000 

0.0000 

0.0000 

O.OObU 

0.0000 

O.OuOO 

0.0000 

0.0000 

ACE 

0.0000 

0,0000 

.  0.0000 

0.0000 

0.0000 

O.oooo 

0.0000 

o.oooo 

bCE 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

ABCE 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

o.oooo 

0.0000 

o.oooo 

OE 

0.0000 

0.0000 

.  0.0000. 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

ADE 

0.0000 

0,0000 

0.0000' 

o.oooo 

U.OGOO 

o.oooo 

0.0000 

0.0000 

eUE 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

0.0000 

0,0000 

0.0000 

ABOE 

0.0000 

0.0000 

0.0000 

o.oooo 

0.0000 

0.0000 

0.0000 

0.0000 

COE 

0.0000 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

ACUE 

0.0000 

0.0000 

0.0000 

o.oooo 

0.0000 

0.0000 

o.oooo 

0.0000 

HCUE 

0.0000 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

0,1)000 

0.0000 

ARCOE 

o.oooo 

0.0000 

o.cooo 

o.oooo 

o.oooo 

0.0000 

o.oooo 

0.0000 

LInEAH 

-O.ObA2 

0«0UU0 

•O.IOBJ 

0.1CB3 

•0,10*3 

•0,1083 

-U.1083 

0.3249 

UUAU 

0.0000 

0,0131 

0.0000 

0.0394 

0.0131 

0.0394 

•0.0131 

0.0919 

CUBIC 

•0.0B24 

0,0000 

-0.16J4 

0.148B 

-0.1S85 

•0.1488 

•0.1S85 

0.3097 

ADDITIVB  CARRYOVER 


AU* 

BO' 

ABD* 

CD* 

ACD' 

BCD' 

ABCD' 

E* 

A 

O.OUOO 

0,0000 

0.0000 

o.cooo 

0.0000 

0.0000 

0.0000 

O.OUOO 

B 

0.0000 

0,0000 

0.0000 

0.0000 

o.uooo 

0.0000 

0.0000 

o.oooo 

AB 

0.0000 

o.oooo 

0.0000 

C.OCOO 

o.oooo 

0.0000 

O.OOUO 

0.0000 

c 

0.0000 

0,0000 

0.0000 

0.0000 

0.0000 

o.oooo 

0,0000 

o.oooo 

AC 

o.oooo 

o.oooo 

0,0000 

0.0000 

o.oouO 

O.UOOO 

o.oooo 

0.0000 

BC 

0.0000 

0.0000 

0.0000 

O.OCOU 

0.0000 

0,0000 

O.UOOO 

O.OOiiO 

ABC 

0.0000 

0.0000 

0.0000 

0.0000 

O.UOOO 

0.0000 

O.OOUO 

0.0000 

0 

•0.2500 

•0,2500 

-0.2500 

-0.2500 

-0.2500 

-0.2500 

•0,2500 

0.0000 

AO 

•O.75O0 

0.250O 

0.2500 

0.2500 

0.2500 

0.2500 

0.2500 

o.ocoo 

80 

0.2500 

0.2500 

-0,7500 

0.2500 

0.2500 

0.2500 

0.2500 

0.0000 

ABO 

•0.2500 

0,7500 

•0.2500 

-0.2500 

-0.2500 

-0,2500 

-0.2500 

0.0000 

CD 

0.2500 

0.2500 

0.2500 

0.7500 

-0.2500 

-0.2SOO 

-0.2500 

0.0000 

ACO 

•0.2500 

•0.2500 

-0.2500 

0.2500 

-0,7500 

0.2500 

0.2500 

0.0000 

BCO 

•0.2500 

•0.2500 

•0,2500 

0.2SU0 

0.2500 

0.2500 

-0.7500 

o.oooo 

ABCD 

0.2500 

0,2500 

0.2500 

•0,2500 

-0.2500 

0.7500 

-0.2500 

o.oooo 

c 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

O.OOUO 

0.0000 

0.8750 

At 

o.oooo 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

o.oooo 

0.1250 

BE 

0.0000 

0.0000 

0.0000 

o.ocoo 

O.UOOO 

o.uooo 

0.0000 

0.1250 

ABE 

0.0000 

0.0000 

0.0000 

o.oooo 

0.0000 

0.0000 

0.0000 

-0.1250 

CE 

0.0000 

0.0000 

0.0000 

O.OCOU 

0.0000 

O.OUOO 

0.0000  ■ 

0.1250 

ACC 

0.0000 

0.0000 

0,0000 

o.ocoo 

o.oooo 

0.0000 

o.oooo 

-0.1250 

BCE 

0.0000 

o.uooo 

0.0000 

o.ocoo 

O.OuOO 

0.0000 

O.OOUO 

-0.1250 

ABCE 

0.0000 

o.oooo 

0,0000 

0.0000 

0.0000 

o.uooo 

0.0000 

0.1250 

DE 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

o.uooo 

0.0000 

0.1250 

ADC 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

o.uooo 

0.0000 

•0.1250 

BDE 

0.0000 

0.0000 

0.0000 

0.0000 

o.uooo 

0.0000 

O.OUOO 

•0.1250 

ABCE 

0.0000 

0.0000 

0.0000 

o.uooo 

O.UOOO 

0.0000 

O.UOOO 

0.1250 

COE 

0.0000 

0,0000 

0.0000 

0.0000 

0.0000 

O.UOOO 

0,0000 

-0.1250 

ACCC 

o.oooo 

o.oooo 

0.0000 

o.ocoo 

0.0000 

0,0000 

O.UOOO 

0.1250 

BCOE 

O.OUOO 

0.0000 

0.0000 

O.OUOO 

0,0000 

O.OUOO 

O.OOUO 

0.1250 

ABCDC 

0.0000 

0.0000 

O.OOuO 

O.OCOU 

O.UOOO 

o.oooo 

0.0000 

-0.1250 

CTKCTir" 

•o.i6»j 

•0.1081 

-0.10*3 

•O.IOhJ 

•0.1083 

•0.1083 

-0.  1083 

0.7541 

OUAO 

0.0394 

0.0919 

•0.0131 

0.1970 

-0.0131 

•0.0131 

-0.0131 

0.197O 

CUBIC 

•0.14*8 

•0.1293 

-0.148* 

•0.0902 

•0,1293 

-0.0902 

-0.16*3 

•0.4610 
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I 

i 


AC 

BE* 

ABC 

h 

Q.UOOO 

0.0000 

0.0000 

8 

o.ooou 

0,0000 

U.OOOO 

A6 

o.ooou 

u.oouo 

O.OOOU 

c 

Q.uouu 

O.UUOQ 

0,0000 

*c 

ObOOOU 

0.0000 

0.0000 

BC 

0.0000 

0.0000 

O.OOOU 

ABC 

o.ooou 

0.0000 

0.0000 

D 

0.0000 

0.0000 

0.0000 

AO 

0.0000 

o.ooou 

0.0000 

150 

o.uooo 

0.0000 

0.0000 

ABU 

0.0000 

o.uooo 

0.0000 

CU 

o.oooo 

o.uuuo 

0.0000 

ACO 

0.0000 

0.0000 

0.0000 

H  BCD 

0.0000 

u.oooo 

0.0000 

ABCO 

c.oooo 

0,0000 

0.0000 

S3  T. 

-0.1250 

-0.1250 

-O.li  0 

S  AE 

-0.8750 

0.1250 

0.1250 

BE 

0.1250 

0.1250 

-0.8750 

ABC 

-0.1250 

0.8750 

-0.1250 

CE 

0.1250 

0.1250 

0.1250 

ACE 

-0.1250 

-0.1250 

-0.1250 

BCE 

-0.1250 

•0.1250 

-0.1250 

Aect 

0.1250 

0.1250 

0.1250 

DC 

0.1250 

0,1250 

0.1250 

AD£ 

-0.1250 

•0.1250 

-0.1250 

Bl)C 

•0,1250 

•0.1250 

-0.1250 

ABDE 

0.1250 

0.1250 

0.1250 

COE 

•0.1250 

•0.1250 

-0.1250 

ACOt 

0.1250 

0.1250 

0.1250 

Bcoe 

0,1250 

0.1250 

0.1250 

A8C0C 

•0.1250 

•0.1250 

-0.1250 

z  lilKkiAH 

•0.1083 

•0.1083 

•0.1083 

U  UOAD 

0.0919 

0.1970 

•0.0131 

f  CUBIC 

•0.1293 

•0.0902 

-0.1293 

CE* 

ACE* 

BCE* 

ABCE* 

DC* 

0.0000 

O.OOuO 

0.0000 

o.ooou 

0.0000 

0.0000 

u.oooo 

o.oouo 

u.oooo 

0.0000 

o.oooo 

0,0000 

o.oouo 

0.0000 

0.0000 

o.ooou 

0.0000 

o.ooou 

O.OuOO 

0.0000 

o.ooou 

o.ooou 

o.oooo 

0.0000 

0.0000 

0.0000 

0.0000 

o.oouo 

0.0000 

0.0000 

u.oooo 

0.0000 

0.0000 

0,0000 

0.0000 

0.0000 

0.0000 

o.oooo 

u.oooo 

0.0000 

0.0000 

o.uooo 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

o.uooo 

0.0000 

o.ocoo 

o.uooo 

0.0000 

0,0000 

o.oooo 

o.oouo 

0.0000 

0.0000 

o.oouo 

o.oouo 

0.0000 

0.0000 

0.0000 

o.ooou 

o.oouo 

0.0000 

0.0000 

o.oouo 

o.oooo 

0.0000 

o.ooou 

0,0000 

0,0000 

u.oooo 

0.0000 

-0.1250 

-0.1250 

-0.1250 

-0.1250 

•0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

0.125O 

0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

-0.1250 

-0.1250 

-0.1250 

-0.1250. 

-0.1250 

0.0250 

-0.3750 

-0.3750 

-0.3750 

0.1250 

0,3750 

-0.6250 

0.3750 

0.3750 

-0.1250 

0.3750 

0.3750 

0.3750 

-0,6250 

-0.1250 

-0.3750 

-0.3750 

0.6250 

-0.3750 

0,1250 

0.1250 

0.1250 

0.1250 

0.1250 

0.8750 

-0.1250 

-0.1250 

-0.1250 

•0.1250 

0.1250 

•0.1250 

•0.1250 

•0.1250 

-0.1250 

0.1250 

0.1250 

0.1250 

.  0.1250 

0.1250 

-  -0.1250 

-0.1250 

-0.1250 

-0.1250 

-0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

0.125O 

•0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

-0.1250 

•a .1250 

-0.1250 

-0.1250 

-0-1250  _ 

0-1250 

•0.1083 

-0.1083 

•0.1083 

•0.1083 

•0.1083 

0.4072 

•0.0131 

-0.0131 

-0.0131 

0.8274 

•0.0122 

•0.0902 

•0.0122 

•0.1683 

0.1439 

r- 
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ADC* 

BUC* 

ABOC* 

CCE* 

acdC 

BCOE* 

ABCUE* 

A 

o.ooou 

0.0000 

o.oouo 

O.OCOO 

O.UOOO 

0,0000 

O.OOOU 

e 

O.OOOU 

O.OOOU 

0.0000 

o.oouo 

O.UOOO 

O.OUOO 

0.0000 

Ai! 

o.oouo 

o.uooo 

0.0000 

O.OOOO 

O.OOOU 

0,0000 

0.0000 

c 

o.uooo 

o.uooo 

0.0000 

o.ooou 

O.OOOU 

0.0000 

O.OOOU 

AC 

O.OUOO 

0,0000 

0.0000 

0.0000 

O.OUOO 

0,0000 

0.0000 

BC 

o.uooo 

0.0000 

0.0000 

o.oooo 

0.0000 

0.0000 

0.0000 

ABC 

o.ooou 

u.oooo 

O.OUOO 

0.0000 

0.0000 

0.0000 

0.0000 

U 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000  . 

0,0000 

AO 

0.0000 

o.oouo 

0.0000 

o.oooo 

0.0000 

0.0000 

0.0000 

HD 

0.0000 

O.OUOO 

O.OUOO 

0.0000 

o.uooo 

0.0000 

0,0000 

ABU 

o.ooou 

o.ooou 

0.0000 

o.ocoo 

o.ooou 

0.0000 

u.oooo 

CU 

o.uooo 

0.0000 

0.0000 

o.oooo 

o.ooou 

u.oooo 

u.oooo 

ACO 

u.oooo 

0.0000 

o.oouo 

o.ocoo 

o.uooo 

o.uooo 

0,0000 

BCD 

0.0000 

O.OUOO 

0.0000 

0.0000 

o.ooou 

O.QUoO 

0,0000 

to 

ABCO 

o.oooo 

O.OUOO 

0.0000 

o.ocoo 

0.0000 

u.oooo 

o.uooo 

0 

E 

-0.1250 

-0.1250 

-0.1250 

•0.1250 

-0,1250 

-0.1250 

-0.1250 

S 

AC 

0.1250 

0.1250 

0.1250 

0.1250 

0,1250 

0.1250 

0.1250 

M 

o 

BC 

0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

ABC 

•0.1250 

•0,1250 

-0.1250 

-0,1250 

-0.1250 

-0.1250 

-0.1250* 

Cb 

0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

ACC 

•0.1250 

•0.1250 

-0.1250 

-0.1250 

-0.1250 

•0.1250 

•0.1250 

BCE 

-0.1250 

-0.1250 

-0.1250 

-0.1250 

-0.1250 

-0.1350 

•0.1250 

ABCE 

0.1250 

0.1250 

0.1250 

0.1250 

0.1250 

0.I25U 

0.1250 

DC 

-0.1250 

-0.1250 

-0.1250 

•0.1250 

•0.1250 

-0.1250 

-0.1250 

ADC 

-0.8750 

0.1250 

0.1250 

0.1250 

0.1250 

0,1250 

0.1250 

BDE 

0.1250 

0.1250 

•0.8750 

0.1250 

0.1250 

0.1250 

0,1250 

ABOC 

-0.1250 

0.8750 

•0.1250 

-0.1250 

-0.1250 

-0.1250 

•0.1250 

COE 

0.1250 

0.1250 

0.1250 

0.6250 

-0,3750 

-0.3750 

-0,3750 

ACDC 

•0.1250 

-0,1250 

•0.1250 

0.3750 

-0.6250 

0.3750 

0»37S0 

BCDE 

•0.1250 

-0.1250 

-0.1250 

0.3750 

0.3750 

0.3750 

-0,6250 

ABCOE 

0.1250 

0.1250 

0.1250 

-0.3750 

-0.3750 

0.6250 

-0.3750 

§ 

L18EAH 

-0.1083 

•0.1083 

•0.1083 

-0.1083 

-0.1083 

•0.1083 

-0.1013 

a 

OUAU 

•0.0131 

•0.0131 

•0.0131 

-0.0131 

•0.0131 

•0.01 31 

-0.0131 

CUBIC 

-0.0123 

0.1439 

•0.1683 

0.45bl 

•0.1683 

-0.1613 

-0.1683 
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MATRIX  c-2*  (see  footnote) 


A 

B 

AB 

c 

AC 

BC 

ABC 

0 

A* 

0*0000 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

B* 

1  *0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

AB* 

•1*0000 

o.uouo 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

C* 

0*S774 

0.5774 

-0.5774 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

AC" 

•0*5774 

•0,5774 

0.5774. 

..  0.0000 

0.0000 

0.0000 

0.0000 

0,0000 

8C" 

0*5774 

-0,5774 

0.5774 

''  0.0000 

o.oooo 

0.0000 

0.0000 

0.0000 

ABC* 

•0,5774 

0,5774 

-0.5774 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0" 

0*3780 

0.3780 

-0.3780 

0.3780 

-0.3780 

-0.3780 

0.3780 

0.0000 

AO" 

•0,3780 

-0.3780 

0.3780 

•0.3780 

0.3780 

0.3780 

-0.3780 

0.0000 

DO" 

0*7746 

-0.2582 

0,2582 

-0.2582 

0.2582 

0.2582 

-0.2582 

0.0000 

ABO* 

•0,774b 

0.2582 

•0,2582 

0.2582 

-0.2582 

-0.2582 

0.2582 

0.0000 

CO" 

0,3780 

0.3780 

•0.3780 

-0.3780 

0.3780 

0,3780 

-0.3780 

0.0000 

ACD" 

•0,3780 

-0,3780 

0.3780 

0.3780 

-0.3780 

•0.3780 

0.3780 

0.0000 

BCD* 

0,7746 

-0.2582 

0.2582 

0.2582 

-0.2582 

-0.2582 

0.2582 

0.0000 

AHCO* 

•0,7746 

0,2582 

•0.2582 

•0.2562 

0.2582 

0.2582 

-0.2582 

0.0000 

E" 

0*2bB2 

0.2582 

•0.2582 

0.2582 

•0.2582 

-0.2582 

0.2582  ' 

0.2582 

AE" 

-0*2582 

-0.2582 

0.2582 

•0.2582 

0.2582 

0.2582 

•0.2582 

•0.2582 

BE* 

0,8814 

-0.1260 

0.1260 

•0.1260 

0.1260 

0.1260 

-0.1260 

-0.1260 

ABC* 

-0.8814 

0.1260 

•0.1260 

0.1260 

-0.1260 

-0.1260 

0.1260 

0.1260 

CC" 

0,4804 

0.4804 

-0.4804 

•0.1601 

0.1601 

0.1601 

-0,1601. 

-0.1601 

ACE" 

•0*4804 

-0,4804 

0.4804 

0.1601 

-0.1601 

-0.1601  , 

0.1601 

0.1601 

BCE* 

0,6742 

-0.4045 

0.4045 

0.1348 

-0.1348 

-0,1348 

0.1348 

0.1348 

ABCC" 

•0,6742 

0.4045 

-0.4045 

•0.1348 

0.1348 

0.1348 

-0.1348 

•0.1348 

DC" 

0,2582 

0.2582 

•0.2582 

0.2582 

-0.2582 

•0.2582 

0.2582 

•0.2582 

AOE* 

•0,2582 

-0.2582 

0.2582 

-0.2582 

0.2582 

0.2582 

-0.2582 

0.2582 

BOE" 

0,8819 

•0.1260 

0.1260 

-0.1260 

0.1260 

0.1260 

-0.1260 

0.1260 

AMOC” 

•0,8819 

0.1260 

•0.1260 

0.1260 

-0,1260 

-0.1260 

0.1260 

-0.1260 

COC" 

0,4804 

0.4804 

-0.4804 

-0.I60I 

0,1601 

0. 1601 

-0.1601 

0.1601 

ACOE" 

•0,4804 

-0.4804 

.  0.4804 

0.1601 

-0.1601 

-0.1601 

0.1601 

-0.1601 

BCCE" 

0*6742 

-0.4045 

0.4045 

0.1348 

-0.1348 

-0.1348 

0.1348 

-0.1348 

AbCOr* 

•0*6742 

0.4045 

•0.4045 

-0.1)48 

0.1348 

0.1 348 

■mnrfiH 

0*0543 

0.IU82 

o.oobo 

0.2166 

0. 01)00 

0,0000 

0.0000 

0.4332 

QUAD 

0*0000 

0.0000 

0,0131 

O.OCOO 

0.0263 

0.0525 

0.0000 

0.0000 

CUBIC 

0*0829 

0.1634 

0,0000 

0.3073 

0.0000 

0.0000 

0.0098 

0.4585 

DIRECT 


AO 

817 

ABO 

CC 

ACU 

BCD 

ABCD 

E 

A* 

0*0000 

0,0000 

O.OCOO 

0.0000 

0.0000 

0.0000 

O.OOUO 

B* 

Q,0OUU 

0,0000 

O.OOUU 

0.0000 

0.0000 

0.0000 

0.0000 

U • 0000 

0,0000 

0,0000 

0,0000 

0.0000 

O.OOuO 

0.0000 

0.0000 

C" 

0,0000 

0,0000 

O.OOOO 

0.0000 

0.0000 

0.0000 

0.0000 

0,0000 

0,0000 

0,0000 

o.ooou 

0.0000 

0.0000 

o.uouo 

0*0000 

0*0000 

0,0000 

0.0000 

o,oouo 

0.0000 

O.DOOO 

0.0000 

0*0000 

ARC" 

0*0000 

o.oooo 

o.oooo 

O.OCOO 

0.0000 

0.0000 

0.0000 

0*0000 

0" 

0,0000 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

0*0000 

AD” 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0001) 

K 

DC* 

0*0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

1 

AHH” 

0,0000 

0.0000 

o.oooo 

0.0000 

o.ooou 

0.0000 

o.ooou 

0.0000 

?? 

CD" 

0*0000 

0.0000 

0.0000 

0.0000 

o.oouo 

0.0000 

0.0000 

0,0000 

ACO" 

O.UOQO 

0.0000 

0.0000 

O.OCOO 

o.oooo 

0.0000 

0.0000 

0,0000 

g 

BCD" 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

o.oooo 

AOCO" 

0*0000 

o.uooo 

0.0000 

0.0000 

0.0000 

0.0000 

o.uoou 

0.0000 

U 

t” 

-0*2$B2 

•Q,2Se2 

0.2582 

-0.2582 

0.2582 

0.2582 

•U.2S82 

o.oouo 

M 

AC* 

0*2502 

0.2582 

•0.2582 

0.2582 

•0.2582 

-0.2582 

0.2582 

0.0000 

BE" 

0,1260 

0,1260 

•0.1260 

0.1260 

•0.1260 

-0.1260 

0.1260 

0.0000 

s 

ABC* 

•0,1260 

•0.1260 

0.1260 

•0.1260 

0.1260 

0,1260 

-0.1260 

o.oouo 

*• 

Ct" 

0* IbOl 

0.1601 

•  0.1601 

0.1601 

-0, 1601 

-0.1601 

0.1601- 

0.0000 

& 

ACE* 

•0,1601 

-0.1601 

0.1601 

•O.ltUl 

0.1601 

0.1601 

-0.1601 

0.0000 

act* 

•0,11AB 

•0.1148 

0.1348 

-0.1348 

0.1348 

0.1348 

-0,1348 

o.oooo 

abce* 

0*1)46 

0.1348 

-0.1348 

0.1348 

>0.1348 

-0.1348 

0.1348 

0.0000 

ot* 

0.2582 

0.2582 

•0.2582 

0.2582 

•0.2582 

-0.2582 

0,2582 

o.oouo 

AOC* 

•0,2562 

-0.2582 

0.2582 

•0.2582 

0.2582 

0.2582 

-0.2582 

0.0000 

BOC* 

•0*1260 

•0.1260 

0.1260 

•0.1260  . 

0.12617 

0.1260 

-0.1260 

0,0000 

ABOC” 

0,1260 

0.1260 

•0.1260 

0.1260 

•0.1260 

•0.12b0 

0.1260 

0.0000 

COC* 

•0*1601 

-0.1601 

0.1601 

•0.1601 

0.1601 

0.1601 

•0.1601 

0.0000 

ACOE* 

0*1601 

0.1601 

•0.1601 

O.UOl 

•0.1601 

•0.1601 

0.1601 

0.0000 

Mrnc* 

0.1148 

0.1348 

•0.1348 

0.1348 

-0.1348 

•0.1348 

0.1341 

o.oouo 

•0.1148 

•0,1349 

0.1348 

•0.1348 

0,1348 

•0,1348 

0.0000 

o 

O.OOuU 

I  t  1 1  f 

HHKlEQilllsH 

UvUgub 

U  •  tf  oo ^ 

s 

QUAD 

0*0525 

O.IOSI 

0.0000 

0.2101 

0.0000 

0,0000 

0.0000 

0.0000 

E 

CUBIC 

0*0000 

0,0000 

0.0195 

'  0.0000 

0.0390 

0,0780 

0.0000 

•0.3317 

•  Th«  Additlv*  v»  Intaraetiva  Carryova  matrix  ia  tha  lama  as  Matrix  C-2  aRCJPt 
fpvpfppd  si^ns  in  sll  columns  vl^ti  sn  odd  number  o*  Is^^srs  In  fcns  isosia 
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AC 

BE 

ABC 

CE 

ACE 

BCE 

ABCE 

BE 

*• 

0.0000 

0.0000 

0.0000 

O.OCOO 

0.0000 
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